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Development  Increases  the  Government's  Financial 
Risk 


Why  GAO  Did  This  Study  What  GAO  Found 


The  Joint  Strike  Fighter  (JSF)  is  the 
Department  of  Defense’s  (DOD) 
most  complex  and  ambitious 
aircraft  acquisition,  seeking  to 
simultaneously  produce  and  field 
three  different  versions  of  the 
aircraft  for  the  Air  Force,  Navy, 
Marine  Corps,  and  eight 
international  partners.  The  total 
investment  required  now  exceeds 
$1  trillion — more  than  $300  billion 
to  acquire  2,456  aircraft  and 
$760  billion  in  life  cycle  operating 
and  support  costs,  according  to 
program  estimates.  The  Ronald  W. 
Reagan  National  Defense 
Authorization  Act  for  Fiscal  Year 
2005  requires  GAO  to  review  the 
JSF  program  annually  for  5  years. 
This  is  the  fifth  and  final  report 
under  the  mandate  in  which  GAO 

(1)  determines  the  program’s 
progress  in  meeting  cost,  schedule, 
and  performance  goals; 

(2)  assesses  manufacturing  results 
and  schedule  risks;  and  (3) 
evaluates  development  test  plans, 
progress,  and  risks.  GAO’s  work 
included  analyses  of  a  wide  range 
of  program  documents,  cost  data 
and  interviews  with  defense  and 
contractor  officials. 


What  GAO  Recommends 


GAO  recommends  that  DOD 
(1)  report  to  the  congressional 
defense  committees  on  the  risks 
and  mitigation  strategy  for  use  of 
cost  reimbursement  contracts  for 
procurement  and  plans  to 
transition  to  fixed  price  contracts 
and  (2)  ensure  that  the  prime 
contractor  performs  periodic 
schedule  risk  analyses  to  improve 
schedule  and  budget  actions.  DOD 
agreed  to  take  these  actions. 

To  view  the  full  product,  including  the  scope 
and  methodology,  click  on  GAO-09-303. 

For  more  information,  contact  Michael  J. 
Sullivan  at  (202)  512-4841  or 
sullivanm@gao.gov. 


JSF  development  will  cost  more  and  take  longer  than  reported  to  the 
Congress  last  year,  and  DOD  wants  to  accelerate  procurement.  Two  recent 
estimates  project  additional  costs  ranging  from  $2.4  billion  to  $7.4  billion  and 
1  to  3  more  years  to  complete  development.  Despite  cost  and  schedule 
troubles,  DOD  wants  to  accelerate  JSF  procurement  by  169  aircraft  from  fiscal 
years  2010  through  2015;  this  could  require  up  to  $33.4  billion  in  additional 
procurement  funding  for  those  6  years.  DOD  plans  to  procure  hundreds  of 
aircraft  on  cost-reimbursement  contracts,  magnifying  the  financial  risk  to  the 
government. 

Ongoing  manufacturing  inefficiencies  and  parts  problems  have  significantly 
delayed  the  delivery  of  test  assets.  The  prime  contractor  has  extended 
manufacturing  schedules  three  times  and  delivered  2  of  13  test  aircraft.  The 
program  is  still  recovering  from  earlier  problems  that  resulted  in  design 
changes,  late  parts  deliveries,  and  inefficient  manufacturing.  The  contractor  is 
taking  positive  steps  to  improve  operations,  the  supplier  base,  and  schedule 
management.  Schedule  risk  analyses  could  further  enhance  management 
insight  into  problem  areas  and  inform  corrective  actions.  Officials  expect  to 
deliver  all  test  aircraft  and  fix  many  problems  by  2010.  By  then,  DOD  plans  to 
have  purchased  62  operational  aircraft  and  will  be  ramping  up  procurement. 
Procuring  large  numbers  of  production  jets  while  still  working  to  deliver  test 
jets  and  mature  manufacturing  processes  does  not  seem  prudent,  and  looming 
plans  to  accelerate  procurement  will  be  difficult  to  achieve  cost  effectively. 

DOD’s  revised  test  plan  adds  a  year  to  the  schedule,  better  aligns  resources 
and  availability  dates,  and  lessens  the  overlap  between  development  and 
operational  testing,  but  it  still  allows  little  time  for  error  discovery  and 
rework.  DOD’s  decision  late  in  2007  to  reduce  test  aircraft  and  flight  tests 
adds  to  risks  while  any  additional  delays  in  delivering  test  aircraft  will  further 
compress  the  schedule.  The  revised  plan  relies  on  state-of-the-art  simulation 
labs,  a  flying  test  bed,  and  desk  studies  to  verify  nearly  83  percent  of  JSF 
capabilities.  Only  17  percent  is  to  be  verified  through  flight  testing.  Despite 
advances,  the  ability  to  so  extensively  substitute  for  flight  testing  has  not  yet 
been  demonstrated.  Significant  overlap  of  development,  test,  and 
procurement  results  in  DOD  making  substantial  investments  before  flight 
testing  proves  that  the  JSF  will  perform  as  expected.  Under  the  accelerated 
procurement  plan,  DOD  may  procure  360  aircraft  costing  an  estimated 
$57  billion  before  completing  development  flight  testing. 


Procurement  Investments  and  Progress  of  Flight  Testing 


2007 

2008 

2009 

2010 

2011 

2012 

2013 

2014 

2015 

Cumulative  procurement 
(billions  of  dollars) 

$0.9 

$3.6 

$7.4 

$15.4 

$28.2 

$42.5 

$57.0 

$72.3 

$89.0 

Cumulative  aircraft 
procured 

2 

14 

30 

62 

132 

241 

360 

506 

684 

Percent  flight  tests 
completed 

<1% 

<1% 

2% 

9% 

34% 

62% 

88% 

100% 

- 

Source:  GAO  analysis  of  DOD  data. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


March  12,  2009 
Congressional  Committees 

The  Joint  Strike  Fighter  (JSF)  is  the  Department  of  Defense’s  (DOD)  most 
complex  and  ambitious  aircraft  acquisition,  seeking  to  simultaneously 
produce  and  field  three  aircraft  variants  for  the  Air  Force,  Navy,  Marine 
Corps,  and  eight  international  partners.  The  JSF  is  critical  to  our  nation’s 
plans  for  recapitalizing  the  tactical  air  forces  and  will  require  a  long-term 
commitment  to  very  large  annual  funding  outlays.  The  total  expected 
investment  is  now  more  than  $1  trillion — more  than  $300  billion  to  acquire 
2,456  aircraft  and  $760  billion  in  life  cycle  operation  and  support  costs, 
according  to  official  program  estimates. 

The  Ronald  W.  Reagan  National  Defense  Authorization  Act  for  Fiscal  Year 
2005  requires  GAO  to  review  the  JSF  program  annually  for  5  years.1 
Previous  reports  identified  opportunities  for  the  program  to  reduce  risks 
and  improve  chances  for  more  successful  outcomes.  We  have  expressed 
concern  about  the  substantial  overlap  of  development,  test,  and 
production  activities  and  recommended  a  more  evolutionary  and 
knowledge-based  acquisition  strategy  with  limited  investment  in 
production  aircraft  until  each  variant  demonstrates  required  capabilities  in 
flight  testing.  Our  most  recent  report  questioned  DOD’s  decision  to  reduce 
test  aircraft  and  flight  hours  and  recommended  that  a  new  comprehensive 
cost  estimate  be  prepared.2  The  department  has  not  implemented  our 
recommendations  and  cost  and  schedule  increases  have  been  the  result. 
This  is  the  fifth  and  final  report  under  the  mandate  in  which  we 
(1)  determine  the  JSF  program’s  progress  in  meeting  cost,  schedule,  and 
performance  goals;  (2)  assess  plans  and  risks  in  manufacturing  and 
capacity  to  accelerate  production;  and  (3)  evaluate  plans,  progress,  and 
risks  with  testing  plans  and  risks  in  testing  activities. 


Tub.  L.  No.  108-375,  §  213  (2004). 

2GAO,  Joint  Strike  Fighter:  Recent  Decisions  by  DOD  Add  to  Program  Risks,  GAO-08-388 
(Washington,  D.C.:  Mar.  11,  2008);  Joint  Strike  Fighter:  Progress  Made  and  Challenges 
Remain,  GAO-07-360  (Washington,  D.C.:  Mar.  15,  2007);  Joint  Strike  Fighter:  DOD  Plans  to 
Enter  Production  before  Testing  Demonstrates  Acceptable  Performance,  GAO-06-356 
(Washington,  D.C.:  Mar.  15,  2006);  and  Tactical  Aircraft:  Opportunity  to  Reduce  Risks  in 
the  Joint  Strike  Fighter  Program  with  Different  Acquisition  Strategy,  GAO-05-271 
(Washington,  D.C.:  Mar.  15,  2005). 
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Results  in  Brief 


To  conduct  this  work,  we  tracked  and  compared  current  cost  and 
schedule  estimates  with  those  of  prior  years,  identified  changes,  and 
determined  causes.  We  obtained  program  status  reports,  manufacturing 
data,  and  test  planning  documents.  We  assessed  the  program’s  scheduling 
estimates  against  best  practices.  We  discussed  results  to  date  and  future 
plans  to  complete  JSF  development  and  accelerate  procurement  with 
DOD,  JSF,  and  contractor  officials.  Some  of  the  cost  data  used  in  our 
report  are  based  on  cost  projections  that  were  current  at  the  time  of  our 
review  rather  than  the  official  program  of  record.  Appendix  I  includes 
additional  information  about  our  scope  and  methodology.  We  conducted 
this  performance  audit  from  June  2008  to  March  2009  in  accordance  with 
generally  accepted  government  auditing  standards.  Those  standards 
require  that  we  plan  and  perform  the  audit  to  obtain  sufficient,  appropriate 
evidence  to  provide  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained 
provides  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our 
audit  objectives. 


JSF  development  will  cost  more  and  take  longer  than  reported  to  the 
Congress  last  year,  and  DOD  wants  to  accelerate  procurement  believing 
that  will  more  quickly  recapitalize  tactical  air  forces.  The  program  office 
estimates  that  an  additional  $2.4  billion  is  needed  for  cost  overruns  on  the 
air  system  and  engine  contracts  and  for  a  1-year  extension  to  the 
development  schedule.  Its  estimate  does  not  include  funding  for  the 
alternate  engine  program.  An  independent  joint  DOD  cost  estimating  team 
identified  a  need  for  as  much  as  $7.4  billion  in  additional  funding  for 
development  through  fiscal  year  2015  and  a  3-year  schedule  extension. 
This  would  increase  total  system  development  costs  to  $51.8  billion — a  17 
percent  increase  from  the  April  2008  estimate — and  delay  completion  of 
development  to  October  2016.  The  joint  team’s  cost  estimate  was  higher 
than  the  program  office’s  estimate  because  it  included  the  alternate  engine 
effort  directed  by  the  Congress  and  made  more  conservative  assumptions 
about  engineer  staffing  levels,  software  requirements  growth, 
manufacturing  labor  hours,  and  flight  testing.  Despite  development  cost 
increases  and  schedule  delays,  DOD  officials  want  to  accelerate  JSF 
procurement  by  purchasing  an  additional  169  aircraft  from  fiscal  years 
2010  through  2015.  This  would  require  up  to  $33.4  billion  in  additional 
procurement  funding  for  those  6  years  and  expose  the  government  to 
additional  risk  from  future  cost  increases  because  of  the  contract  type. 

The  plan  would  not  increase  the  total  JSF  quantity  through  completion, 
but  would  buy  these  aircraft  earlier  than  planned.  DOD  did  not  estimate 
the  net  effect  this  plan  would  have  on  future  procurement  funding  to 
complete  JSF  acquisition. 
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Ongoing  manufacturing  inefficiencies  and  parts  problems  have 
significantly  delayed  the  delivery  of  needed  test  aircraft,  and  the  prime 
contractor  has  not  yet  achieved  the  levels  of  learning  expected,  even  as 
the  program  ramps  up  production.  The  contractor  has  extended 
manufacturing  schedules  three  times  and  produced  2  of  13  development 
test  aircraft.  In  the  past  year,  budgeted  hours  to  complete  work  on  all  test 
aircraft  have  increased  by  40  percent.  The  program  is  still  recovering  from 
earlier  problems  in  development — late  release  of  engineering  drawings 
resulting  in  design  changes  and  delays  in  establishing  the  supplier  base, 
late  part  deliveries,  and  inefficient  manufacturing  line  work-arounds 
where  unfinished  work  is  completed  out  of  station.  The  prime  contractor 
has  taken  steps  to  improve  manufacturing,  the  supplier  base,  and  schedule 
management.  A  thorough  schedule  risk  analysis  could  further  enhance 
management  insight  into  areas  of  uncertainty  and  better  inform 
subsequent  actions  to  correct  persistent  problems,  such  as  schedule 
slippage  and  allocation  of  management  reserves.  Officials  expect  to  deliver 
all  test  aircraft  and  address  most  of  these  problems  by  2010.  By  that  time, 
DOD  plans  to  have  procured  62  operational  aircraft  and  will  be  quickly 
ramping  up  production.  As  DOD  has  a  large  backlog  of  production  jets  on 
order  and  is  still  working  to  deliver  test  jets  and  mature  manufacturing 
processes,  plans  to  accelerate  procurement  will  be  difficult  to  achieve  in  a 
cost-effective  manner. 

DOD’s  revised  test  plan  adds  an  extra  year  to  the  schedule,  better  aligns 
resources  and  availability  dates,  and  lessens  the  overlap  between 
development  and  operational  testing,  but  it  is  still  aggressive  and  allows 
little  time  for  error  discovery,  rework,  and  recovery  from  downtime. 

DOD’s  decision  late  in  2007  to  reduce  the  number  of  development  test 
aircraft  and  to  decrease  the  number  of  flight  tests  added  risk,  while  any 
additional  delays  in  delivering  test  aircraft  will  further  compress  the 
schedule.  The  independent  cost  team  believes  flight  testing  will  require  an 
additional  2  years  to  complete  and  suggests  more  flight  hours  to  test 
mission  systems  and  the  carrier  variant  in  particular.  The  revised  plan 
relies  on  advanced  and  robust  simulation  labs,  a  flying  test  bed,  and 
analytical  studies  to  verify  nearly  83  percent  of  the  aircraft’s  capabilities 
while  only  17  percent  is  to  be  verified  through  flight  testing.  While  the  labs 
appear  more  prolific,  integrated,  and  capable  than  those  used  in  legacy 
programs,  the  ability  to  substitute  for  flight  testing  has  not  yet  been 
demonstrated.  Significant  overlap  of  development,  test,  and  production 
schedules  results  in  DOD  making  substantial  investments  before  flight 
testing  proves  that  the  JSF  will  perform  as  expected.  Under  the 
accelerated  procurement  plan,  DOD  may  procure  360  aircraft  costing  an 
estimated  $67  billion  before  completing  development  flight  testing. 
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Acquiring  large  numbers  of  aircraft  before  testing  successfully 
demonstrates  that  the  design  is  mature,  meets  performance  requirements, 
and  is  suitable  could  result  in  substantial  future  cost  growth  to  correct 
deficiencies  found  during  testing. 

To  enhance  congressional  oversight  and  increase  the  likelihood  of  more 
successful  program  outcomes,  we  are  recommending  that  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  report  on 
the  JSF’s  contracting  strategy  to  the  congressional  defense  committees  by 
October  1,  2009.  This  report  would  include  (1)  an  explanation  of  the 
remaining  program  risks  and  the  factors  justifying  the  continued  use  of 
cost  reimbursement  contracts  for  JSF’s  future  low-rate  production 
quantities,  (2)  the  program’s  strategy  for  managing  and  mitigating  risks 
associated  with  the  use  of  cost  contracts  versus  fixed-price  contracts,  and 
(3)  plans  for  transitioning  to  fixed-price  contracts  for  production, 
including  time  frames  and  criteria.  To  further  maintain  confidence  that  the 
program  is  on  track  to  meet  planned  cost,  schedule,  and  performance 
goals,  we  are  also  recommending  that  the  JSF  Program  Office  ensure  that 
the  prime  contractor  performs  periodic  schedule  risk  analyses  for  the  JSF 
program  to  provide  better  insight  into  management  reserve,  production 
efficiencies,  and  schedule  completion. 


Background 


The  JSF  is  a  joint,  multinational  acquisition  program  for  the  Air  Force, 
Navy,  and  Marine  Corps  and  eight  international  partners.  The  program 
began  in  November  1996  with  a  5-year  competition  between  Lockheed 
Martin  and  Boeing  to  determine  the  most  capable  and  affordable 
preliminary  aircraft  design.  Lockheed  Martin  won  the  competition,  and  the 
program  entered  system  development  and  demonstration  in  October  2001. 
Program  goals  are  to  develop  and  field  an  affordable,  highly  common 
family  of  stealthy,  next-generation  strike  fighter  aircraft  for  the  Navy,  Air 
Force,  Marine  Corps,  and  U.S.  allies. 

The  JSF  is  a  single-seat,  single-engine  aircraft,  designed  to  rapidly 
transition  between  air-to-ground  and  air-to-air  missions  while  still 
airborne.  To  achieve  its  mission,  the  JSF  will  incorporate  low-observable 
(stealth)  technologies,  defensive  avionics,  advanced  onboard  and  offboard 
sensor  fusion,  internal  and  external  weapons,  and  advanced  prognostic 
maintenance  capability.  The  JSF  family  consists  of  three  variants.  The 
conventional  takeoff  and  landing  (CTOL)  variant  will  primarily  be  an  air- 
to-ground  replacement  for  the  Air  Force’s  F-16  Falcon  and  the  A-10 
Warthog  aircraft,  and  will  complement  the  F-22A  Raptor.  The  short  takeoff 
and  vertical  landing  (STOVL)  variant  will  be  a  multi-role  strike  fighter  to 
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replace  the  Marine  Corps’  F/A-18C/D  Hornet  and  AV-8B  Harrier  aircraft. 
The  carrier-suitable  variant  will  provide  the  Navy  a  multi-role,  stealthy 
strike  aircraft  to  complement  the  F/A-18E/F  Super  Hornet.  DOD  is 
planning  to  buy  a  total  of  2,456  JSFs  and  allies  are  expected  to  procure  a 
minimum  of  730  CTOL  and  STOVL  aircraft. 

Because  of  the  program’s  sheer  size  and  the  numbers  of  aircraft  it  will 
replace,  the  JSF  is  the  linchpin  of  DOD’s  long-term  plan  to  modernize 
tactical  air  forces.  It  is  DOD’s  largest  acquisition  program,  with  total  cost 
currently  estimated  at  $300  billion,  and  the  longest  in  planned  duration, 
with  procurement  projected  through  2034.  In  addition,  the  JSF  remains 
DOD’s  largest  cooperative  program.3  Our  international  partners  are 
providing  about  $4.8  billion  toward  development,  and  foreign  firms  are 
part  of  the  industrial  base  producing  aircraft.  DOD’s  funding  requirements 
for  the  JSF  assume  economic  benefits  from  these  foreign  purchases  in 
reducing  unit  costs  for  U.S  aircraft. 

Table  1  shows  the  evolution  of  DOD’s  official  estimated  cost,  quantity,  and 
deliveries  from  the  initiation  of  system  development  in  October  2001  to  the 
current  official  program  of  record  dated  December  2007  and  submitted  to 
the  Congress  in  April  2008.  It  depicts  quantities  reduced  in  the  last  major 
program  restructure  in  2004,  the  impacts  of  increased  costs  on  unit  prices, 
and  the  slip  in  delivering  initial  operating  capability  to  the  warfighter. 


3The  international  partners  are  the  United  Kingdom,  Italy,  the  Netherlands,  Turkey,  Canada, 
Australia,  Denmark,  and  Norway.  These  nations  are  contributing  funds  for  system 
development  and  have  signed  agreements  to  procure  a  minimum  of  730  aircraft.  Israel  and 
Singapore  are  security  cooperation  participants,  and  several  other  nations  have  reportedly 
expressed  interest  in  acquiring  aircraft. 
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Table  1 :  Changes  in  JSF  Program  Cost,  Quantity,  and  Delivery  Estimates 

October  2001 


(system 

development  December  2003 


start) 

(2004  replan) 

December  2006 

December  2007 

Expected  quantities 

Development  quantities 

14 

14 

15 

13 

Procurement  quantities  (United  States  only) 

2,852 

2,443 

2,443 

2,443 

Total  quantities 

2,866 

2,457 

2,458 

2,456 

Cost  estimates  (then-year  dollars  in  billions) 

Development 

$34.4 

$44.8 

$44.2 

$44.4 

Procurement 

196.6 

199.8 

255.1 

254.0 

Military  construction 

2.0 

0.2 

0.5 

0.5 

Total  program  acquisition 

$233.0 

$244.8 

$299.8 

$298.9 

Unit  cost  estimates  (then-year  dollars  in  millions) 

Program  acquisition 

$81 

$100 

$122 

$122 

Average  procurement 

69 

82 

104 

104 

Estimated  delivery  dates 

First  operational  aircraft  delivery 

2008 

2009 

2010 

2010 

Initial  operational  capability 

2010-2012 

2012-2013 

2012-2015 

2012-2015 

Source:  GAO  analysis  of  DOD  data. 

Notes:  Data  are  from  the  annual  selected  acquisition  reports  that  are  dated  in  December  of  each  year 
but  are  not  officially  released  and  reported  to  the  Congress  until  March  or  April  of  the  following  year. 
The  December  2003  data  reflect  the  last  major  restructuring  of  the  program.  The  December  2007 
data  represent  the  official  program  of  record  at  the  time  of  our  review  and  was  reported  to  the 
Congress  in  April  2008. 

Military  construction  costs  have  not  been  fully  established,  and  the  reporting  basis  changed  over  time 
in  these  DOD  reports. 

In  our  March  2008  report,4  we  stated  that  JSF  costs  would  likely  grow 
much  higher  than  reported  because  the  program  of  record  at  that  time  did 
not  include  all  acquisition  costs  (including  the  alternate  engine  program 
directed  by  the  Congress),  made  overoptimistic  assumptions,  and  did  not 
fully  reflect  the  mounting  cost  and  schedule  pressures  from  manufacturing 


4GAO-08-388. 
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inefficiencies  and  compressed  time  frames  for  completing  development. 
We  questioned  the  Mid-Course  Risk  Reduction  Plan  adopted  by  DOD  in 
September  2007  that  cut  two  development  test  aircraft,  reduced  test 
flights,  and  accelerated  the  reduction  in  the  prime  contractor’s 
development  workforce  in  order  to  replenish  management  reserves 
depleted  by  design  changes  and  manufacturing  problems.  We 
recommended  that  DOD  accomplish  a  full  and  comprehensive  estimate  of 
the  total  program  verified  by  an  independent  third  party  and  revisit  the 
Mid-Course  Risk  Reduction  Plan  with  an  intensive  analysis  of  the  causes 
of  management  reserve  depletion,  progress  against  the  baseline 
manufacturing  schedule,  and  correction  of  deficiencies  in  the  contractor’s 
earned  value  management  system.  DOD  agreed  to  make  a  comprehensive 
independent  cost  estimate,  but  decided  to  go  ahead  as  planned  with  the 
Mid-Course  Risk  Reduction  Plan,  stating  that  it  would  monitor  and 
evaluate  progress  and  revise  the  plan  later  if  it  failed  to  achieve 
expectations. 


More  Money  and  Time 
Will  Be  Needed  to 
Complete  JSF 
Development,  While 
DOD  Plans  to 
Accelerate 
Procurement 


Two  recent  estimates  indicate  that  JSF  development  will  cost  more  and 
take  longer  to  complete  than  reported  to  the  Congress  in  April  2008, 
primarily  because  of  contract  cost  overruns  and  extended  time  to 
complete  flight  testing.  DOD  also  plans  to  accelerate  aircraft  procurement 
over  the  next  6  years — buying  more  aircraft  sooner  than  planned  last  year. 
This  new  plan  will  require  significantly  more  procurement  funding  sooner, 
but  officials  did  not  assess  its  net  effect  on  total  program  costs  through 
completion  of  JSF  acquisition. 


New  Estimates  Project 
Rising  Costs  and  Further 
Delays  to  Complete  JSF 
Development 


Development  costs  are  projected  to  increase  between  $2.4  billion  and 
$7.4  billion  and  the  schedule  for  completing  system  development  extended 
from  1  to  3  years,  according  to  recent  estimates — one  by  the  JSF  Program 
Office  and  one  by  a  joint  team  of  Office  of  the  Secretary  of  Defense  (OSD), 
Air  Force,  and  Navy  officials.5  Cost  overruns  on  both  the  aircraft  and 
engine  contracts,  delays  in  manufacturing  test  aircraft,  and  a  need  for  a 


5In  April  2008,  OSD  asked  the  Cost  Analysis  Improvement  Group  (CAIG)  to  lead  a  joint 
estimating  team  in  assessing  the  overall  executability  of  the  JSF  development  program  and 
resource  requirements  for  fiscal  years  2010  through  2015.  The  joint  estimating  team  was 
composed  of  CAIG,  Air  Force,  and  Navy  representatives  as  well  as  subject  matter  experts. 
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longer,  more  robust  flight  test  program  are  the  primary  cost  drivers.  The 
joint  team’s  estimate  is  higher  than  the  program  office’s  because  it 
included  costs  for  the  alternate  engine  program  directed  by  the  Congress 
and  used  more  conservative  assumptions  based  on  current  and  legacy 
aircraft  experiences.  Program  officials  contend  that  funding  the  program 
to  the  higher  cost  estimate  is  premature  and  believe  processes  are  in  place 
to  substantially  improve  on  the  test  experiences  of  past  programs. 
Regardless,  both  estimates  agree  that  cost  and  time  to  complete 
development  have  increased  from  the  official  program  of  record  at  the 
time  of  our  review.  (See  table  2.) 


Table  2:  Estimated  Cost  and  Schedule  for  System  Development 

2007  program  of  record 

JSF  Program  Office 

Joint  estimating  team 

Development  costs  to  complete 

$7.4  billion 

$9.8  billion 

$14.8  billion5 

Total  development  costs 

$44.4  billion 

$46.8  billion 

$51.8  billion5 

Date  to  complete  development 

October  2013 

October  2014 

October  2016 

Source:  GAO  analysis  of  DOD  data. 


The  joint  estimating  team  only  projected  costs  through  fiscal  year  2015.  Extending  development  to 
October  2016,  as  the  team  projects,  would  increase  both  cost  figures.  DOD  data  suggest  that  1  year 
of  additional  flight  testing  and  other  government  costs  could  be  about  $700  million. 

The  program  office’s  revised  development  cost  estimate  projects  an 
additional  $2.4  billion  and  a  1-year  extension  in  the  schedule  compared  to 
the  official  program  of  record  reported  to  the  Congress.  This  would 
increase  the  system  development  portion  of  the  acquisition  program  to 
$46.8  billion  and  delay  its  completion  to  October  2014.  The  cost  increases 
primarily  resulted  from  the  following  factors. 

•  $1.2  billion  for  aircraft  development.  Program  officials  declared  a  cost 

overrun  on  the  prime  air  system  contract  because  of  increased  labor 
hours,  higher  prices,  and  supply  shortages.  Included  in  this  figure  is 
$200  million  to  be  added  to  the  contractor’s  management  reserve.6  Last 
year,  we  reported7  that  mounting  cost  and  schedule  pressures  depleted 
reserves  much  faster  than  anticipated.  By  summer  2007,  the  program 


^Management  reserves  are  budgeted  funds  set  aside  for  unanticipated  development 
challenges  and  increase  a  program’s  capacity  to  deal  with  unknowns. 

7GAO-08-388. 
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had  spent  about  two-thirds  of  budgeted  funds  but  had  accomplished 
only  half  the  work  required.  Since  then,  DOD’s  efforts  to  restore 
reserves  and  fix  manufacturing  inefficiencies  have  not  fully  achieved 
intended  results,  requiring  another  cash  infusion. 

•  $800  million  for  engine  development.  According  to  officials  from  the 
Defense  Contract  Management  Agency,  the  engine  contractor 
continued  to  face  development  problems,  which  resulted  in  a  contract 
cost  overrun.  Higher  costs  for  labor  and  materials,  supplier  problems, 
and  the  rework  needed  to  correct  deficiencies  with  the  engine  blade 
design  discovered  during  ground  testing  were  major  contributing 
factors. 

•  $300  million  for  flight  test  extension.  The  program  extended  system 
development  by  1  year  to  provide  more  time  for  development  and 
operational  flight  testing.  In  April  2008,  an  operational  test  review  team 
recommended  extending  the  development  contract  by  1  year.  The 
review  team  considered  but  dismissed  several  other  options  to  address 
the  schedule  problem,  including  deferring  requirements. 

On  the  other  hand,  the  joint  estimating  team  estimates  that  it  will  cost 
$14.8  billion  to  complete  JSF  development,  $7.4  billion  more  than  the 
official  program  of  record  at  the  time  of  our  review.  This  would  increase 
the  total  development  costs  to  $51.8  billion  from  the  $44.4  billion  reported 
to  the  Congress  last  year — a  17  percent  increase.  The  joint  team  also 
projected  a  3-year  program  extension  beyond  the  program  of  record  in 
order  to  complete  system  development,  2  years  more  than  the  new 
program  office  estimate. 

The  joint  team’s  estimate  was  $5  billion  more  than  the  new  program  office 
estimate.8  Several  factors  account  for  the  difference  between  the  two 
estimates. 

•  Alternate  engine.  The  joint  estimating  team  included  $1.4  billion  to 
complete  development  of  an  alternate  engine  for  the  JSF;  the  program 
office  estimate  did  not  include  alternate  engine  costs.  The  Congress 
has  directed  DOD  to  develop  a  second  source  for  the  JSF  engine  to 


sThe  difference  in  the  two  estimates  is  actually  more  than  $5  billion.  The  joint  estimating 
team’s  numbers  only  go  through  fiscal  year  2015,  but  they  are  expecting  development  to 
take  at  least  1  more  year  beyond  then.  Program  data  presented  to  the  Joint  Chiefs  suggest 
that  an  additional  year  for  development  flight  testing  and  other  government  and  contractor 
expenses  could  cost  about  $700  million. 
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induce  competition  and  to  ensure  that  one  engine’s  failures  would  not 
ground  all  JSFs,  thereby  reducing  operational  risks  in  the  future.  DOD 
has  not  wanted  to  pursue  this  second  engine  source  and  twice  removed 
funding  from  the  JSF  program  line. 

•  Engineering  staffing.  The  joint  estimating  team  projected  a  need  for 
the  contractor  to  retain  considerably  more  engineering  staff  and  for 
longer  periods  of  time  than  the  program  office  estimate  to  complete 
development,  evaluate  test  results,  and  correct  deficiencies.  Releasing 
engineering  staff  prematurely  risks  not  discovering  problems  until  late 
in  development  or  during  fielding,  when  they  would  be  more  expensive 
to  address. 

•  Software  development.  The  joint  estimating  team  believes  that  the 
software  productivity  rate  will  be  less  than  the  program’s  calculation 
and  anticipates  much  more  growth  in  software  requirements.  The  JSF 
aircraft  is  expected  to  require  7.5  million  lines  of  computer  code — the 
most  by  far  of  any  aircraft.  By  comparison,  the  F/A-18E/F  has  only  1.1 
million  and  the  F-22A  2.2  million.  Experiences  on  past  acquisitions 
have  shown  30  to  100  percent  growth  in  software  requirements  over 
time,  while  the  JSF  Program  Office  estimate  assumed  no  growth. 

•  Flight  testing.  The  joint  estimating  team  projects  that  flight  testing  will 
require  more  time  and  effort  than  the  program  office  has  built  into  its 
current  schedule.  Continuing  delays  in  delivering  test  aircraft  are 
expected  to  hamper  and  further  compress  test  schedules.  In  particular, 
the  joint  team  projects  that  the  two  aircraft  dedicated  to  carrier 
suitability  tests  will  be  late  off  the  production  line,  thereby  delaying 
test  activities.  It  also  projected  that  the  JSF  will  require  about  2,700 
hours  of  flight  testing  for  mission  systems,  significantly  more  than  the 
1,700  hours  that  the  program  office  currently  estimates. 

•  Manufacturing  production  hours.  The  joint  estimating  team  projects 
that  production  span  times  for  the  JSF  will  be  longer  than  the  program 
office  estimates  based  on  the  program’s  performance  to  date  and 
experience  of  recent  programs,  such  as  the  F/A-18  E/F  and  the  F-22A. 
The  span  time  is  an  indicator  of  how  long  the  manufacturing  effort 
takes  and  when  flight  testing  can  begin.  The  program  office  assumes 
that  span  times  will  decrease  over  the  course  of  the  development 
contract.  We  note  that  span  times  typically  increase  during 
development,  as  was  the  case  for  both  the  B-2  bomber  and  F-22A 
programs. 

Program  officials  believe  that  their  estimate  is  more  accurate  and  that 

providing  extra  funding  to  address  future  risks  is  premature  and  does  not 
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provide  incentives  for  contractors  to  control  costs.  The  program  office 
attributes  its  lower  cost  estimate  to  several  factors.  First,  the  quantity, 
quality,  and  flexibility  of  the  JSF  laboratories  should  enable  the  program 
to  reduce  more  risks  in  a  laboratory  environment,  rather  than  through 
flight  testing,  which  is  considerably  more  expensive.  In  addition,  the 
program’s  efforts  to  develop  the  final  software  system  infrastructure  early 
should  reduce  significant  software  problems  later  in  the  program, 
according  to  the  program  office.  The  program  office  also  believes  that 
costs  will  be  lower  because  progress  in  several  key  development  areas  is 
either  matching  plans  or  is  ahead  of  where  legacy  programs  were  at 
similar  points  of  time  in  their  development.  For  example,  the  program  is 
currently  reducing  engineering  staff  as  planned.  The  program  is  also 
producing  software  at  a  rate  significantly  higher  than  that  of  the  F-22A 
program  and  is  at  least  18  months  ahead  of  where  the  F-22A  program  was 
at  a  similar  point  in  developing  mission  systems,  according  to  program 
officials.  Officials  told  us  that  they  intend  to  fund  the  fiscal  year  2010 
development  budget  based  on  the  joint  team’s  higher  estimate.  However,  it 
is  not  clear  at  this  stage  which  estimate  will  serve  as  the  basis  for  future 
budget  submissions. 


Much  Higher  Annual 
Procurement  Funding 
Required  to  Accelerate  JSF 
Procurement 


The  program  office  and  joint  estimating  team  also  projected  procurement 
funding  requirements  for  the  6-year  period  fiscal  years  2010-2015  based  on 
DOD  plans  to  accelerate  procurement  of  operational  aircraft.  Through  this 
effort,  DOD  wants  to  recapitalize  tactical  air  forces  sooner  and  mitigate 
projected  fighter  shortfalls  in  the  future.  Compared  to  last  year,  this 
accelerated  plan  would  procure  an  additional  169  aircraft  during  these  6 
years,  moving  aircraft  that  had  been  scheduled  for  procurement  beyond 
2015  to  earlier  years.  According  to  the  two  estimates,  this  plan  would 
require  from  $21.8  billion  to  $33.4  billion  more  funding  than  the  official 
program  of  record,  as  shown  in  table  3. 


Table  3:  Projected  Procurement  Funding  Requirements  for  Fiscal  Years  2010  through  2015 

2007  program  of  record 

JSF  Program  Office 

Joint  estimating  team 

Procurement  funding  requirements 

$59.7  billion 

$81.5  billion 

$93.1  billion 

Procurement  quantity 

485 

654 

654 

Average  procurement  unit  cost 

$123  million 

$125  million 

$142  million 

Source:  GAO  analysis  of  DOD  data. 


The  joint  team’s  estimate  is  higher  than  the  program  office’s,  primarily  for 
these  reasons: 
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•  The  joint  team  projected  slower  gains  in  production  efficiency  than  the 
program  office.  Typically,  production  efficiency  is  improved  and  unit 
costs  are  lowered  over  time  as  a  workforce  becomes  more  experienced 
building  a  new  product  and  manufacturing  processes  are  honed. 

•  The  joint  team  also  assumed  fewer  savings  from  commonality. 
Commonality — a  key  selling  point  for  the  JSF  program — refers  to  the 
use  of  the  same  or  similar  parts,  structures,  and  subsystems  shared  by 
the  three  variants.  Greater  commonality  can  save  money  by  decreasing 
development  times  and  facilitating  economic  order  quantities. 

•  The  team  projected  higher  labor  and  material  costs  and  longer 
production  span  times,  based  on  JSF  performance  to  date  in 
manufacturing  development  test  aircraft. 

Table  4  shows  the  additional  aircraft  and  funding  requirements  for  DOD’s 
accelerated  plan  compared  to  the  official  program  of  record.  These 
quantities  are  for  the  United  States  only;  during  this  same  period,  the 
international  partners  are  expected  to  buy  273  aircraft. 


Table  4:  Proposed  Accelerated  Procurement 


FY10 

FY11 

FY12 

FY13 

FY14 

FY15 

Total 

2007  program  of  record  funding  requirements3 

$6.0 

$6.9 

$10.5 

$10.8 

$12.4 

$13.1 

$59.7 

2007  program  of  record  quantities 

30 

43 

82 

90 

110 

130 

485 

Program  office  accelerated  funding  requirements3 

$8.0 

$12.8 

$14.3 

$14.4 

$15.3 

$16.7 

$81.5 

J  ointteam  accelerated  funding  requirements3 

$8.9 

$13.9 

$16.7 

$16.6 

$17.8 

$19.3 

$93.1 

Accelerated  quantities 

32 

70 

109 

119 

146 

178 

654 

Source:  GAO  analysis  of  DOD  data. 


■Funding  requirements  are  expressed  in  billions  of  then-year  dollars. 


Procurement  of 
Operational  Aircraft  under 
Cost  Reimbursement 
Contracts  to  Continue; 
Increases  the 

Government’s  Exposure  to 
Risks 


The  JSF  program  is  procuring  a  substantial  number  of  aircraft  on  cost 
reimbursement  contracts.  Cost  contracts  place  most  of  the  risk  on  the 
buyer — DOD  in  this  case — who  is  liable  to  pay  more  than  budgeted  should 
labor,  material,  or  other  incurred  costs  be  more  than  expected  when  the 
contract  was  signed.  JSF  officials  plan  to  procure  at  least  the  first  four 
low-rate  production  lots  under  cost  reimbursement  contracts  and  to 
transition  to  fixed-price  instruments  when  appropriate,  possibly  between 
lots  5  and  7  (fiscal  years  2011  to  2013).  It  is  unclear  exactly  how  and  when 
this  will  happen,  but  the  expectation  is  to  transition  to  fixed  pricing  once 
the  air  vehicle  has  a  mature  design,  has  been  demonstrated  in  flight  tests, 
and  is  producible  at  established  cost  targets.  To  date,  DOD  has  procured 
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the  first  three  lots  for  a  total  of  30  aircraft  and  $7.4  billion  on  cost 
reimbursement  terms.  Under  the  accelerated  procurement  plan,  DOD 
could  procure  as  many  as  360  aircraft  costing  about  $57  billion  through 
fiscal  year  2013  on  cost  reimbursement  contracts,  as  illustrated  in  figure  1. 


Figure  1:  Cumulative  Procurement  Costs  and  Quantities 

Cumulative  cost _  Cumulative  quantity 


Dollars  (in  billions) 

100 

Period  of  possible 


Number  of  aircraft 
800 


Source:  GAO  analysis  of  DOD  data. 


Cost  reimbursement  contracts  provide  for  payment  of  allowable  incurred 
costs,  to  the  extent  prescribed  in  the  contract.  According  to  the  Federal 
Acquisition  Regulation,  cost  reimbursement  contracts  are  suitable  for  use 
only  when  uncertainties  involved  in  contract  performance  do  not  permit 
costs  to  be  estimated  with  sufficient  accuracy  to  use  any  type  of  fixed- 
price  contract.9  Cost  reimbursement  contracts  for  weapon  production  are 
considered  appropriate  when  the  program  lacks  sufficient  knowledge 
about  system  design,  manufacturing  processes,  and  testing  results  to 
establish  firm  prices  and  delivery  dates.  In  contrast,  a  fixed-price  contract 
provides  for  a  pre-established  price,  places  more  of  the  risk  and 
responsibility  for  costs  on  the  contractor,  and  provides  more  incentive  for 
efficient  and  economical  performance. 

Procuring  up  to  360  production  aircraft  on  cost  reimbursement 
contracts — nearly  15  percent  of  the  total  DOD  program — seems  to  be  a 
tacit  acknowledgment  by  DOD  and  the  contractor  that  knowledge  on  JSF 


BFederal  Acquisition  Regulation  §  16.301-2. 
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design,  production  processes,  and  costs  for  labor  and  material  is  not  yet 
sufficiently  mature  and  that  pricing  information  is  not  exact  enough  for 
the  contractor  to  assume  the  risk  under  a  fixed-price  contract.  It  also 
seems  to  be  a  consequence  of  the  substantial  concurrency  of  development, 
test,  and  production  built  into  the  JSF  schedule.  Significant  overlap  of 
these  activities  means  that  DOD  is  procuring  considerable  quantities  of 
operational  aircraft  while  development  test  aircraft  are  still  on  the 
production  line  and  much  in  advance  of  testing  to  prove  aircraft 
performance  and  suitability.  Establishing  a  clear  and  accountable  path  to 
ensure  that  the  contractor  assumes  more  of  the  risk  is  prudent.  We  note 
that  the  significant  ramp  up  in  JSF  production  under  the  accelerated 
profile  starts  with  lot  5,  the  fiscal  year  2011  procurement  of  70  aircraft. 


Continued 
Manufacturing 
Inefficiencies  Will 
Make  It  Difficult  for 
the  Program  to  Meet 
Its  Production 
Schedule 


Manufacturing  of  development  test  aircraft  is  taking  more  time,  money, 
and  effort  than  planned,  but  officials  believe  that  they  can  work  through 
these  problems  and  deliver  the  11  remaining  test  aircraft  by  early  2010. 
However,  by  that  time,  DOD  may  have  procured  as  many  as  62  production 
aircraft,  accumulating  a  backlog  of  aircraft  to  be  produced.  Manufacturing 
inefficiencies  and  parts  shortages  continue  to  delay  the  completion  and 
delivery  of  development  test  aircraft  needed  for  flight  testing.  The 
contractor  has  not  yet  demonstrated  mature  manufacturing  processes,  or 
an  ability  to  produce  at  currently  planned  annual  rates.  It  has  taken  steps 
to  improve  manufacturing,  the  supplier  base,  and  schedule  management. 
However,  given  the  manufacturing  challenges,  we  believe  that  DOD’s  plan 
to  accelerate  production  in  the  near  term  adds  considerable  risk  and  will 
be  difficult  to  achieve  in  a  cost-effective  manner. 


Time  and  Money  Needed 
for  Manufacturing 
Development  Test  Aircraft 
Continue  to  Increase 


The  prime  contractor  has  restructured  the  JSF  manufacturing  schedule 
three  times,  each  time  lengthening  the  time  to  deliver  aircraft  to  the  test 
program.  Delays  and  manufacturing  inefficiencies  are  prime  causes  of 
contract  cost  overruns.  The  contractor  has  produced  two  development 
test  aircraft — an  original  non-production  representative  model  and  the 
first  STOVL  aircraft.  It  now  projects  delivering  the  remaining  11  aircraft  in 
2009  and  early  2010.  Problems  and  delays  are  largely  the  residual  effects 
from  difficulties  early  in  development.  The  effects  of  the  late  release  of 
engineering  drawings,  design  changes,  delays  in  establishing  a  supplier 
base,  and  parts  shortages  continue  to  cause  delays  and  force  inefficient 
production  line  work-arounds  where  unfinished  work  is  completed  out  of 
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station.10  Data  provided  by  the  Defense  Contract  Management  Agency  and 
the  JSF  Program  Office  show  continuing  critical  parts  shortages,  out-of- 
station  work,  and  quality  issues.  The  total  projected  labor  hours  to 
manufacture  test  aircraft  increased  by  40  percent  just  in  the  past  year,  as 
illustrated  in  figure  2. 


Figure  2:  JSF  Labor  Hour  Estimates  for  Development  Test  Aircraft 

Budgeted  labor  hours  (in  thousands) 

400 


358 


Aircraft 

Budgeted  hours  (J  une  2007) 
Budgeted  hours  (J  une  2008) 
Source:  GAO  analysis  of  DOD  data. 


Performance  data  for  two  major  cost  areas — wing  assembly  and  mate  and 
delivery — indicate  even  more  substantial  growth.  Figure  3  compares  the 
increased  budgeted  hours  in  the  new  schedule  to  last  year’s  estimates.  The 
2007  schedule  assumed  a  steeper  drop  in  labor  hours  as  more  units  are 
produced  and  manufacturing  and  worker  knowledge  increases;  this 
increased  efficiency  because  of  learning  is  typical  of  production  programs. 
The  new  schedule,  based  upon  actual  performance,  projects  a  less  steep 


10An  efficient  production  line  establishes  an  orderly  flow  of  work  as  a  product  moves  from 
workstation  to  workstation  and  on  to  final  assembly.  Out-of-station  work,  sometimes 
referred  to  as  traveled  work,  refers  to  completing  unfinished  work  on  major  components, 
for  example,  the  wings,  after  they  have  left  the  wing  workstation  and  moved  down  the 
production  line  to  another  station,  such  as  mate  and  final  assembly. 
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decline  in  labor  hours,  indicating  lesser  gains  in  worker  efficiency.  As  of 
June  2008,  the  planned  hours  for  these  two  major  stations  increased  by 
about  90  percent  over  the  June  2007  schedule,  which  itself  had  shown  an 
increase  from  the  2006  schedule. 


Figure  3:  Budgeted  Manufacturing  Hours  by  Development  Aircraft  (Wing  and  Mate  and  Delivery  Stages) 
Wing  manufacturing  data  Mate  and  delivery  manufacturing  data 
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Source:  GAO  analysis  ofDOD  data. 


The  overlap  in  the  work  schedule  between  manufacturing  the  wing  and 
mating  (connecting)  it  to  the  aircraft  fuselage  has  been  a  major  concern 
for  several  years  because  it  causes  inefficient  out-of-station  work.  The 
contractor  continues  to  address  this  concern,  but  the  new  schedule 
indicates  that  this  problem  will  continue  at  least  through  2009.  One 
indicator  of  its  persistence  is  the  projected  hours  for  building  the  last  test 
aircraft.  As  figure  4  shows,  estimated  labor  hours  increased  more  than  80 
percent  from  the  June  2007  to  June  2008  schedules,  and  the  planned  hours 
for  wing  assembly  and  for  the  mate  and  delivery  phases  more  than  tripled. 
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Figure  4:  Comparison  of  Budgeted  Labor  Hours  to  Build  Last  Development  Aircraft 
June  2007  June  2008 
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Source:  GAO  analysis  of  DOD  data. 


Prime  Contractor  Actions 
to  Improve  Schedule 
Management, 
Manufacturing  Efficiency, 
and  Supplier  Base 


Our  evaluation  determined  that  the  prime  contractor  now  has  good  tools 
and  integrated  processes  in  place  that  should  improve  its  schedule 
management  activities  and  is  also  implementing  actions  to  improve 
manufacturing  efficiency,  the  delivery  of  parts,  and  proactive  oversight  of 
subcontractors.  The  effects  from  these  recent  actions  are  not  yet  fully 
apparent,  and  the  contractor  has  not  yet  accomplished  its  own  schedule 
risk  assessment  that  could  provide  more  insight  into  impacts  from  areas  of 
risk  and  uncertainty.  The  coming  year  will  be  critical  for  implementing 
management  improvements  in  order  to  accomplish  a  firm  and  effective 
transition  from  manufacturing  a  few  test  aircraft  to  producing  operational 
aircraft  at  high  annual  rates. 
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Prime  Contractor’s  Scheduling 
Management  Processes  Meet 
Many  Best  Practices,  but 
Program  Risks  Are  Not  Entirely 
Visible 


Improvements  in 
Manufacturing  and  Supplier 
Base  Are  Ongoing 


The  prime  contractor  demonstrated  to  us  that  its  schedule  management 
processes  largely  meet  established  best  practices  criteria.  With 
improvements  implemented  in  2008,  the  contractor’s  management  systems 
meet  or  partially  meet  eight  of  the  nine  established  criteria.11  For  example, 
the  master  schedule  can  identify  and  track  activities  associated  with  over 
600  projects.  It  also  establishes  the  critical  path  between  activities, 
allowing  the  program  to  examine  the  impacts  of  schedule  delays  and 
determine  schedule  flexibility.  Appendix  II  discusses  our  examination  of 
the  prime  contractor’s  schedule  management  process  against  best 
practices  criteria  in  more  detail. 

The  one  area  not  meeting  best  practices  was  related  to  performing  a 
schedule  risk  analysis,  and  as  a  result,  the  contractor  has  limited  insight 
into  areas  of  risk  and  uncertainty  in  the  schedule.  The  prime  contractor 
has  not  conducted  its  own  risk  assessment  that  would  (1)  determine  the 
level  of  confidence  it  has  in  meeting  completion  dates  and  (2)  identify  and 
apportion  reserve  funds  for  contingencies.  A  thorough  risk  analysis  could 
improve  management  insight  and  subsequent  corrective  actions  on  two 
recurring  problem  areas  in  particular:  schedule  slippage  and  inadequate 
management  reserve  levels.  Naval  Air  Systems  Command  officials  did 
accomplish  an  independent  schedule  risk  analysis  that  indicated  that  the 
program  could  slip  more  than  2  years  based  on  the  productivity  risks 
associated  with  software  development  and  assembly  of  the  various 
airframes  as  well  as  the  time  needed  to  complete  ah  flight  testing.  Both  the 
contractor  and  the  JSF  Program  Office  disputed  the  findings  of  the  Naval 
Air  Systems  Command  schedule  risk  analysis  primarily  because  the 
analysis  was  done  without  direct  involvement  of  program  officials. 

The  prime  contractor  is  implementing  changes  designed  to  address  the 
manufacturing  inefficiencies  and  parts  shortages  discussed  earlier.  These 
include  (1)  increasing  oversight  of  key  subcontractors  that  are  having 
problems,  (2)  securing  long-term  raw  material  purchase  price  agreements 
for  both  the  prime  and  key  subcontractors,  and  (3)  implementing  better 
manufacturing  line  processes.  On  this  latter  point,  according  to  program 
officials,  the  prime  contractor  has  taken  specific  steps  to  improve  wing 
manufacturing  performance — one  of  the  most  troublesome  workstations. 
Defense  Contract  Management  Agency  officials  noted  that  the  contractor 


1  'GAO,  GAO  Cost.  Estimating  and  Assessment  Guide:  Best  Practices  for  Developing  and 
Managing  Capital  Program  Costs,  GAO-09-3SP  (Washington,  D.C.:  March  2009).  Also,  see 
app.  II  for  our  assessment  of  the  prime  contractor’s  schedule  management  processes. 


Page  18 


GAO-09-303  Joint  Strike  Fighter 


produced  the  second  STOVL  aircraft  variant  with  less  work  performed  out 
of  station  than  for  the  first  STOVL  aircraft.  Also,  program  office  and 
contractor  officials  report  some  alleviation  of  parts  shortages  and 
improvements  in  quality,  but  also  believe  that  the  effects  from  previous 
design  delays,  parts  shortages,  and  labor  inefficiencies  will  continue  to 
persist  over  the  near  term. 

The  lag  time  between  taking  action  and  demonstrating  improvement  may 
partly  explain  why  some  manufacturing  performance  metrics  are  not 
demonstrating  a  clear  continued  rate  of  quality  improvement,  as  would  be 
desirable  and  expected  for  a  program  ramping  up  annual  production  rates. 
This  lag  time  may  be  evident  in  two  important  metrics — scrap,  rework, 
and  repair  rates  and  manufacturing  defect  rates — both  of  which  have 
increased  somewhat  since  2006.  Program  and  contracting  officials  point 
out,  however,  that  while  this  performance  is  not  desirable,  these  and  other 
metrics  compare  very  favorably  with  those  of  prior  acquisitions  at  similar 
stages  of  development,  including  the  F-16  and  F-22A. 

Supplier  costs  are  expected  to  make  up  an  even  more  substantial  share  of 
total  expenses  as  the  program  moves  further  into  production.  According  to 
contractor  officials,  efforts  are  focused  on  maturing  the  supply  base  and 
working  more  closely  and  directly  with  key  suppliers  to  reduce  costs, 
alleviate  parts  shortages,  and  support  higher  production  rates.  Key 
suppliers  have  struggled  to  develop  critical  and  complex  parts  while 
others  have  had  problems  with  limited  production  capacity.  For  example, 
the  supplier  responsible  for  the  advanced  electro-hydraulic  actuation 
system  had  delivered  parts  with  missing  subcomponents  and  parts  that 
were  not  built  to  specifications.  The  major  “team  mate”  supplier 
responsible  for  fuselage  and  tail  assembly  has  experienced  delays  caused 
by  its  limited  machining  capacity.  Given  these  supplier  issues, 
manufacturing  inefficiencies,  and  accumulating  backlog  in  production,  we 
believe  that  the  program’s  plans  to  accelerate  production  in  the  near  term 
adds  considerable  risk  and  will  be  difficult  to  achieve  in  a  cost-effective 
manner. 
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The  JSF’s  Test  Plan  Is 
Improved  but  Faces 
Numerous  Challenges 
to  Complete 
Development  on  Time 
and  on  Budget 


DOD  will  make  significant  investments — in  both  dollars  and  the  number  of 
aircraft  procured — before  completing  JSF  flight  testing.  DOD’s  proposal  to 
accelerate  procurement  further  increases  financial  risks  in  a  very 
challenging  test  environment.  DOD’s  new  test  plan  adds  an  extra  year  to 
the  schedule  and  better  aligns  resources,  but  is  still  aggressive  with  little 
room  for  error,  presenting  a  formidable  challenge  to  complete  system 
development,  support  initial  operational  testing,  and,  eventually,  a  full  rate 
production  decision.  DOD  decisions  to  reduce  development  test  aircraft 
and  flight  tests  add  to  the  risks,  while  any  additional  delays  in 
manufacturing  test  aircraft  will  further  compress  the  schedule.  The  JSF 
has  just  begun  development  flight  testing  with  two  test  aircraft  and  has 
already  experienced  some  setbacks — normal  in  any  program,  but  of 
special  concern  when  assets  are  minimal.  Some  in  DOD  forecast  that 
another  2  or  more  years  beyond  the  1-year  extension  just  approved  will 
eventually  be  needed  to  successfully  prove  aircraft  performance  and 
complete  system  development.  The  department  has  stated  that  the 
contractor’s  state-of-the-art  ground  test  labs  and  a  flying  test  bed  will 
mitigate  risks  in  the  flight  regimen  and  their  use  will  effectively  substitute 
for  flight  testing.  This  approach  is  promising,  but  not  yet  proven. 


Significant  Investments  DOD  is  investing  heavily  in  procuring  JSF  aircraft  before  flight  testing 
Made  before  Development  proves  that  they  will  perform  as  expected.  Procuring  aircraft  before 
Flight  Tests  Are  Completed  testing  successfully  demonstrates  that  the  design  is  mature  and  that  the 

weapon  system  will  work  as  intended  increases  the  likelihood  of  design 
and  requirements  changes  resulting  in  subsequent  cost  growth,  schedule 
delays,  and  performance  limitations.  Also,  systems  already  built  and 
fielded  may  later  require  substantial  modifications,  further  adding  to  costs. 
The  uncertain  environment  as  testing  progresses  is  one  reason  why  the 
prime  contractor  and  DOD  are  using  cost  reimbursable  contracts  until 
rather  late  in  procurement.  Table  5  depicts  planned  investments — in  both 
dollars  and  aircraft — prior  to  the  completion  of  development  flight  testing. 
Under  the  accelerated  production  plan  and  using  the  lower  procurement 
cost  estimate  prepared  by  the  program  office,  DOD  may  procure  360 
aircraft  at  a  total  estimated  cost  of  $67  billion  before  development  flight 
testing  is  completed.  This  overlap  will  be  further  exacerbated  should  the 
joint  estimating  team’s  predictions  of  higher  cost  and  lengthier  schedule 
prove  accurate. 
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Table  5:  Overlap  of  Procurement  Investments  and  Flight  Testing 


2007 

2008 

2009 

2010 

2011 

2012 

2013 

2014 

Cumulative  procurement 
(billions  of  then-year  dollars) 

$0.9 

$3.6 

$7.4 

$15.4 

$28.2 

$42.6 

$57.0 

$72.3 

Cumulative  quantity 

2 

14 

30 

62 

132 

241 

360 

506 

Contract  type 

Cost 

Cost 

Cost 

Cost 

Cost  or 
fixed 

Cost  or 
fixed 

Cost  or 
fixed 

Fixed 

Percentage  of  flight  test 
program  completed 

<1% 

<1% 

2% 

9% 

34% 

62% 

88% 

100% 

LIMITED  KNOWLEDGE  GAINED  FROM  FLIGHT  TESTING  ►  MORE  KNOWLEDGE  GAINED  FROM 


FLIGHT  TESTING 


Source:  GAO  analysis  of  DOD  data. 

Notes:  Flight  testing  data  reflect  the  percentage  of  total  development  flight  tests  completed  at  the  time 
of  the  planned  investment  decision,  which  is  expected  in  J  anuary  of  each  year. 


Flight  Testing  Is  Still  in  Its  The  JSF  program  had  completed  about  2  percent  of  its  development  flight 
Infancy  and  Has  Fallen  testing  as  of  November  2008.  Figure  5  shows  the  expected  ramp  up  in 

Behind  Schedule  flight  testing  with  most  effort  occurring  in  fiscal  years  2010  through  2012. 

Past  programs  have  shown  that  many  problems  are  not  discovered  until 
flight  testing.  As  such,  the  program  is  likely  to  experience  considerable 
cost  growth  in  the  future  as  it  steps  up  its  flight  testing,  discovers  new 
problems,  and  makes  the  necessary  technical  and  design  corrections. 
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Figure  5:  JSF  Planned  Development  Test  Flights 
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Source:  GAO  analysis  of  DOD  data. 


While  the  program  has  been  able  to  demonstrate  basic  aircraft  flying 
capabilities,  it  has  recently  experienced  testing  delays  and  has  fallen 
behind  the  flight  test  plan  established  in  2007.  At  the  time  of  our  review, 
the  program  had  flown  about  half  of  its  planned  155  flight  tests  for  2008. 
The  test  program  currently  has  two  development  test  aircraft  and  an 
integrated  airborne  test  bed,  with  the  following  experiences  to  date: 

•  Sixty-five  test  flights  on  the  original,  non-production  representative 
prototype  contributed  to  discoveries  in,  among  other  things,  landing 
gear  door  fitting,  aerial  refueling  operations,  and  weapons  bay 
functions.  The  prototype  experienced  a  3-month  delay  because  of 
engine  bay  nacelle  vent  fan  malfunctions  that  were  subsequently 
resolved. 

•  Initial  testing  of  the  first  of  12  production  representative  prototypes 
began  in  June  2008 — a  STOVL  variant  flown  in  conventional  mode.  By 
the  time  of  our  review  it  accumulated  14  test  flights  demonstrating 
important  handling  qualities  and  reducing  risks  associated  with,  among 
other  things,  the  landing  gear,  fuel  system  performance,  and  STOVL 
doors  operation.  Engine  problems  delayed  full  STOVL  testing  by  6 
months. 
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•  Thirty-seven  flights  on  the  cooperative  avionics  test  bed  tested  mission 
system  software  and  demonstrated  communication  and  navigation 
capabilities. 

Looking  ahead,  the  program  expects  to  take  delivery  of  the  remaining 
development  test  aircraft  during  2009  and  early  in  fiscal  year  2010.  In  the 
same  time  frame,  it  also  plans  to  begin  flight  testing  6  of  its  12  production 
representative  prototype  test  aircraft  (CTOL  and  STOVL  aircraft), 
including  the  first  2  aircraft  dedicated  to  mission  system  testing.  The  first 
carrier  variant  development  test  aircraft  is  expected  to  begin  full  flight 
testing — including  ship  suitability  testing — in  2010.  A  fully  integrated, 
mission  capable  aircraft  is  not  expected  to  enter  flight  testing  until  2012  by 
which  time  DOD  plans  to  have  already  purchased  241  aircraft  for  about 
$43  billion  under  cost  reimbursement  contracts. 


Program’s  Test  Plan 
Extends  Development  and 
Relies  Heavily  on  Ground 
Testing  and  Simulations  to 
Verify  Aircraft 
Performance 


The  JSF  Program  Office  developed  a  new  and  improved  test  plan  in  the 
spring  of  2008  that  extended  the  development  period  by  1  year,  better 
aligned  test  resources  and  availability  dates,  and  lessened  the  overlap 
between  development  and  operational  testing.  The  new  plan  is  still 
aggressive,  however,  and  has  little  room  for  error  discovery,  rework,  and 
recovery  from  downtime  should  test  assets  be  grounded  or  otherwise 
unavailable.  The  sheer  complexity  of  the  JSF  program — with  7.5  million 
lines  of  software  code,  three  variants,  and  multi-mission  development — 
suggests  that  the  aircraft  will  encounter  many  unforeseen  problems  during 
flight  testing  requiring  additional  schedule  to  rework.  Our  past  work  has 
shown  that  programs  that  do  not  allow  sufficient  time  to  address  the 
inevitable  problems  discovered  in  testing,  run  a  greater  risk  of  significant 
cost  increases  and  schedule  delays  when  problems  do  arise.  The  joint 
estimating  team  noted  that  the  program’s  flight  test  plan  assumed  a  higher 
productivity  than  has  been  seen  on  recent  flight  test  programs.  As  such, 
the  joint  team  believes  that  flight  testing  will  require  an  additional  2  years 
to  complete  beyond  the  1  year  already  added  to  development  and  suggests 
that  more  flight  test  hours  will  be  necessary  to  test  mission  systems  and 
the  carrier  variant  in  particular. 


The  Mid-Course  Risk  Recovery  Plan,  approved  in  late  2007,  cut  two 
development  test  aircraft,  reduced  test  flights,  and  relies  more  on  ground 
laboratories  and  simulations  to  verify  performance,  adding  substantial  risk 
to  the  program.  Our  2008  report12  discussed  the  objections  from  several 


12GAO-08-388. 
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prominent  DOD  offices  to  the  Mid-Course  Risk  Recovery  Plan.  The 
Director  of  Operational  Test  and  Evaluation,  for  example,  identified  risks 
in  the  revised  verification  strategy  and  cited  inadequate  capacity  to  handle 
the  pace  of  mission  testing  and  ship  suitability  and  signature  testing.  This 
increases  the  likelihood  of  not  finding  and  resolving  critical  design 
deficiencies  until  operational  testing,  when  it  is  more  costly  and  disruptive 
to  do  so. 

The  test  plan  relies  heavily  on  a  series  of  advanced  and  robust  simulation 
labs  and  a  flying  test  bed  to  verify  aircraft  and  subsystem  performance. 
Figure  6  shows  that  83  percent  of  the  aircraft’s  capabilities  are  to  be 
verified  through  labs,  the  flying  test  bed,  and  subject  matter  analysis,  while 
only  17  percent  of  test  points  are  to  be  verified  through  flight  testing.  The 
JSF  program  spent  $5  billion  on  its  system  of  simulation  labs  and  models. 
Program  officials  argue  that  their  heavy  investment  in  simulation  labs  will 
allow  early  risk  reduction,  thereby  reducing  the  need  for  additional  flight 
testing,  controlling  costs,  and  meeting  the  key  milestones  of  the  program’s 
aggressive  test  plan. 


Figure  6:  Breakdown  of  Verification  Venues  for  the  JSF 


The  JSF  program’s  simulation  labs  appear  more  prolific,  integrated,  and 
capable  than  the  labs  used  in  past  aircraft  programs.  The  program  utilizes 
18  labs  for  development  whereas  only  9  were  used  for  the  F-22A,  7  for  the 
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F-18,  and  5  for  the  F-16.  According  to  program  officials,  the  greater 
number  of  labs  allows  engineers  to  work  simultaneously  on  different 
development  blocks,  reducing  bottlenecks  that  may  occur  in  testing.  In 
contrast,  engineers  of  the  F-18  and  F-22A  programs  had  to  interrupt  or 
shut  down  development  on  one  development  block  while  they  were 
making  corrections  to  another.  Also  in  contrast  to  past  programs,  key  JSF 
simulation  labs  are  colocated  at  a  Lockheed  Martin  plant  in  Fort  Worth, 
Texas.  The  F-22A  program  utilized  three  locations  and  two  different 
companies.  According  to  the  program  office,  the  colocation  of  the  key 
testing  labs  facilitates  more  seamless  integration  of  key  aircraft 
components.  Program  officials  also  noted  that  JSF  labs  use  the  actual 
aircraft  components  to  a  greater  extent  than  labs  did  in  past  programs  and 
also  have  greater  software  processing  capacity.  This  allows  for  more 
realistic  data,  which  should  reduce  the  need  for  additional  flight  testing. 
Further,  the  JSF  utilizes  the  first  fully  integrated  airborne  test  bed  for 
mission  system  testing.  According  to  program  officials,  the  test  bed’s 
design  is  geospatially  proportionate  to  an  actual  F-35  aircraft,  enhancing 
its  ability  to  accurately  verify  aircraft  performance. 

While  the  labs  appear  more  prolific,  integrated,  and  complex  than  those 
used  in  legacy  programs,  concerns  about  their  extensive  use  in  verifying 
aircraft  performance  remain.  The  extent  of  the  JSF  program’s  planned  lab 
use  is  unprecedented,  but  the  ability  to  substitute  for  flight  testing  has  not 
yet  been  demonstrated.  In  addition,  the  labs  have  yet  to  be  fully 
accredited.  Accreditation  is  required  to  ensure  the  adequate  capability  of 
labs  and  models  to  perform  verification  tasks.  It  is  critical  that  the  models 
behave  like  aircraft  to  ensure  that  the  system’s  performance  requirements 
are  being  verified  accurately.  The  program  office  said  that  it  is  on  track  to 
complete  the  accreditation  of  the  labs  in  time  to  begin  verifying  system 
performance.  However,  the  Director  of  Operational  Test  and  Evaluation 
reports  that  the  progress  of  the  accreditation  support  packages  is  behind 
schedule  and  suggests  that  more  flight  testing  may  be  needed  as  the 
accreditation  process  reveals  the  limitations  of  the  models.  Some  DOD 
officials  are  also  concerned  that  the  labs  will  be  understaffed.  The  Director 
of  Operational  Test  and  Evaluation  and  DOD’s  joint  estimating  team  both 
reported  that  the  program’s  current  resource  plans  reduce  engineering 
staff  too  rapidly.  Engineering  and  test  personnel  are  critical  to  analyzing 
the  data  generated  from  the  labs.  Without  adequate  staff,  there  is  a  greater 
risk  that  the  labs  will  not  be  sufficiently  utilized,  which  could,  in  turn, 
result  in  schedule  delays  or  cost  increases. 
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Early  Operational 
Assessment  Identifies 
Several  Challenges  That 
May  Have  Operational 
Impact  If  Not  Addressed 


While  the  program  is  projecting  that  it  will  meet  all  key  performance 
parameters,  most  will  not  be  verified  through  ground  and  flight  testing 
until  fiscal  years  2010  through  2013.  In  addition,  a  2008  operational 
assessment  by  the  Air  Force  Operational  Test  and  Evaluation  Center 
pointed  out  several  technical  challenges  that  it  believes  are  likely  to  have  a 
severe  operational  impact  if  not  adequately  addressed.  While  some  of  the 
report’s  concerns  are  not  specific  requirements,  some  DOD  officials 
believe  that  the  shortfalls  may  adversely  affect  the  JSF’s  ability  to  meet 
warfighter  needs.  For  example: 


•  The  current  F-35  power  system  may  cause  excessive  damage  to  runway 
surfaces  which  could  limit  its  ability  to  operate  in  certain  locations. 

The  program  is  still  evaluating  the  problem  and  plans  to  gather  data 
and  conduct  further  studies  when  full-scale  models  or  actual  aircraft 
are  available.  According  to  a  program  official,  changes  to  the  aircraft’s 
design  or  to  the  current  concept  of  operations  may  be  needed.  The 
program  has  alerted  the  services  and  believes  it  will  have  a  better 
understanding  of  the  problem  sometime  in  2009. 

•  Thermal  management  challenges  hamper  the  ability  to  conduct 
missions  in  hot  and  cold  environments.  The  Director  of  Operational 
Test  and  Evaluation  reported  that  an  alternative  main  engine  fuel  pump 
to  remedy  this  problem  is  under  development  but  will  not  be  available 
before  the  low-rate  initial  production  Lot  3,  which  is  likely  to  affect 
operational  testing.  The  test  team  aborted  a  test  sortie  because  of  high 
fuel  temperatures  in  June. 

•  The  JSF’s  advanced  integrated  support  system  aims  to  improve  and 
streamline  aircraft  logistics  and  maintenance  functions  in  order  to 
reduce  life  cycle  costs.  The  current  integrated  support  system  for  the 
JSF  prohibits  operating  two  detachments  from  one  squadron 
simultaneously.  This  limitation  will  severely  affect  current  operating 
practices,  which  include  dividing  one  squadron  of  aircraft  into 
subgroups  to  deploy  and  operate  at  different  locations. 


Conclusions 


The  JSF  is  DOD’s  largest  and  most  complex  acquisition  program  and  the 
linchpin  of  efforts  to  recapitalize  our  tactical  air  forces.  It  is  now  in  its 
most  challenging  phase,  at  a  crossroads  of  a  sort.  Challenges  are  many — 
continuing  cost  and  schedule  pressures;  complex,  extensive,  and  unproven 
software  requirements;  and  a  nascent,  very  aggressive  test  program  with 
diminished  flight  test  assets.  Looking  forward,  the  contractor  plans  to 
complete  work  expeditiously  to  deliver  the  test  assets,  significantly  step 
up  flight  testing,  begin  verifying  mission  system  capabilities,  mature 
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manufacturing  processes,  and  quickly  ramp  up  production  of  operational 
aircraft.  As  such,  the  credibility  of  the  program’s  test  plans,  manufacturing 
capacity,  and  subsequent  cost  and  schedule  estimates  should  become 
more  apparent. 

The  program  must  move  forward,  but  given  all  these  challenges, 
accelerating  procurement  in  a  cost-reimbursement  contract 
environment — where  uncertainties  in  contract  performance  do  not  permit 
costs  to  be  estimated  with  sufficient  accuracy  to  use  any  type  of  fixed- 
price  contract — places  very  significant  financial  risk  on  the  government. 
Accelerating  plans  also  does  not  equate  to  an  ability  to  deliver  to  those 
plans.  Because  the  program’s  manufacturing  processes  are  still  maturing 
and  flight  testing  is  still  in  its  infancy,  incorporating  an  accelerated 
production  schedule  introduces  even  more  risk  and  uncertainty  to  the 
program.  Our  past  work  has  shown  that  programs  that  make  production 
decisions  prior  to  fully  proving  a  system’s  design  through  testing  and 
demonstration  of  mature  manufacturing  processes  have  an  increased  risk 
of  design  and  production  changes  and  retrofits  of  completed  aircraft.13 
Until  the  contractor  demonstrates  that  it  can  produce  aircraft  in  a  timely 
and  efficient  manner,  DOD  cannot  fully  grasp  future  funding  requirements. 
DOD  needs  tangible  assurance  from  the  prime  contractor  that  it  can  meet 
expected  development  and  production  expectations.  By  accelerating 
low-rate  production  quantities  before  manufacturing  and  testing  processes 
are  mature,  DOD  accepts  most  of  the  contractors’  production  and 
manufacturing  inefficiencies.  At  minimum,  the  contractor  needs  to 
develop  a  detailed  plan  demonstrating  how  it  can  successfully  meet 
program  development  and  production  goals  in  the  near  future  within  cost 
and  schedule  parameters.  With  an  improved  contracting  framework  and  a 
more  reasoned  look  to  the  future,  the  JSF  program  can  more  effectively 
meet  DOD  and  warfighter  needs  in  a  constrained  budget  environment. 


Recommendations 
Executive  Action 


for  Given  the  program’s  ongoing  manufacturing  problems  and  nascent  flight 
test  program,  we  believe  that  moving  forward  with  an  accelerated 
procurement  plan  is  very  risky.  This  risk  is  reflected  by  the  extended  use 
of  cost  reimbursement  contracts  for  low-rate  production  quantities — a 
contract  mechanism  that  places  most  of  the  cost  risk  on  the  government. 


13GAO’s  past  work  has  shown  that  manufacturing  maturity  is  achieved  when  a  company 
can  produce  a  product  within  cost,  schedule,  and  quality  targets.  A  best  practice  is  to 
ensure  that  all  key  manufacturing  processes  are  repeatable,  sustainable,  and  capable  of 
consistently  producing  parts  within  the  product’s  quality  tolerances  and  standards. 
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As  such,  to  enhance  congressional  oversight,  increase  the  likelihood  of 
more  successful  program  outcomes,  and  maintain  confidence  that  the 
program  is  on  track  to  meet  planned  cost,  schedule,  and  performance 
goals,  we  recommend  that  the  Secretary  of  Defense  take  the  following  two 
actions: 


Agency  Comments 
and  Our  Evaluation 


1.  Direct  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  to  report  to  the  congressional  defense  committees  by 
October  1,  2009.  This  report  should  include,  at  minimum, 

•  an  explanation  of  the  cost  and  other  risks  associated  with  a  cost- 
reimbursable  contract  as  compared  to  a  fixed-price  contract  for 
JSF’s  future  low-rate  production  quantities, 

•  the  program’s  strategy  for  managing  and  mitigating  risks  associated 
with  the  use  of  cost  contracts,  and 

•  plans  for  transitioning  to  fixed-price  contracts  for  production  to 
include  time  frames  and  criteria. 

2.  Direct  the  JSF  Program  Office  to  ensure  that  the  prime  contractor 
performs  periodic  schedule  risk  analyses  for  the  JSF  program  to 
provide  better  insight  into  management  reserve,  production 
efficiencies,  and  schedule  completion  to  allow  for  corrections  as  early 
as  possible. 


DOD  provided  us  with  written  comments  on  a  draft  of  this  report.  The 
comments  are  reprinted  in  appendix  III. 

DOD  substantively  agreed  with  our  first  recommendation  regarding  a 
report  to  the  Congress  on  contracting  strategy,  but  believed  that  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  should  be 
responsible  for  the  report,  not  the  JSF  Program  Office  as  stated  in  our 
draft.  As  the  milestone  decision  authority,  the  Under  Secretary  is 
responsible  for  approving  the  contracting  strategy,  contract  awards,  and 
the  transition  to  full  rate  production.  We  agree  with  DOD  and  revised  the 
recommendation  accordingly. 

DOD  also  agreed  with  the  second  recommendation  and  will  direct  that  the 
prime  contractor  perform  periodic  schedule  risk  analysis.  In  coordination 
with  the  JSF  Program  Office,  the  department  intends  to  determine  an 
optimum  schedule  for  the  contractor  that  will  provide  insight  into  JSF  cost 
and  schedule  to  influence  key  milestones  and  decision  making. 
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We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense;  the 
Secretaries  of  the  Air  Force,  Army,  and  Navy;  and  the  Director  of  the 
Office  of  Management  and  Budget.  The  report  also  is  available  at  no 
charge  on  the  GAO  Web  site  at  http://www.gao.gov. 

If  you  or  your  staff  have  any  questions  concerning  this  report,  please 
contact  me  at  (202)  512-4841  or  sullivanm@gao.gov.  Contact  points  for  our 
Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on  the 
last  page  of  this  report.  Staff  members  making  key  contributions  to  this 
report  are  listed  in  appendix  IV. 


Michael  J.  Sullivan 
Director 

Acquisition  and  Sourcing  Management 
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List  of  Congressional  Committees 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Daniel  K.  Inouye 
Chairman 

The  Honorable  Thad  Cochran 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Ike  Skelton 
Chairman 

The  Honorable  John  M.  McHugh 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  John  P.  Murtha 
Chairman 

The  Honorable  C.W.  Bill  Young 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Appendix  I:  Scope  and  Methodology 


To  determine  the  Joint  Strike  Fighter  (JSF)  program’s  progress  in  meeting 
cost,  schedule,  and  performance  goals,  we  received  briefings  by  program 
and  contractor  officials  and  reviewed  financial  management  reports, 
budget  documents,  annual  selected  acquisition  reports,  monthly  status 
reports,  performance  indicators,  and  other  data.  We  compared  reported 
progress  with  prior  years’  data,  identified  changes  in  cost  and  schedule, 
and  obtained  officials’  reasons  for  these  changes.  We  interviewed 
Department  of  Defense  (DOD),  JSF  Program  Office,  and  contractor 
officials  to  obtain  their  views  on  progress,  ongoing  concerns  and  actions 
taken  to  address  them,  and  future  plans  to  complete  JSF  development  and 
accelerate  procurement.  This  review  was  the  fifth  and  final  effort  under 
the  mandate  of  the  Ronald  W.  Reagan  National  Defense  Authorization  Act 
for  Fiscal  Year  2005.  We  were  provided  sufficient  information  to  make  the 
assessments  contained  in  this  report. 

In  assessing  program  cost  estimates,  we  compared  the  official  program 
cost  estimate  in  the  December  31,  2007,  selected  acquisition  report  to 
estimates  developed  by  the  JSF  program  and  an  independent  joint 
estimating  team  for  fiscal  years  2010  through  2015.  Because  the  fiscal  year 
2010  budget  had  not  been  submitted  to  the  Congress  at  the  time  of  the 
draft  report,  some  of  the  report’s  findings  are  based  on  preliminary  cost 
projections  that  existed  at  the  time  of  our  review  rather  than  the  official 
program  of  record.  As  such,  the  cost  projections  in  this  report  may  be 
different  than  the  final  fiscal  year  2010  program  of  record.  We  interviewed 
members  of  the  joint  estimating  team  to  obtain  a  detailed  understanding  of 
the  methodology,  data,  and  approach  used  in  developing  their  cost 
estimate  and  schedule  risk  analysis  of  the  JSF  program.  We  also 
interviewed  JSF  program  officials  to  understand  the  program’s  cost 
estimate  methodology,  assumptions,  and  results  and  to  obtain  their 
response  to  the  joint  estimating  team’s  analysis.  Based  on  this  analysis,  we 
were  able  to  identify  significant  differences  in  the  cost  estimating 
methodologies  and  assumptions  used  by  the  joint  estimating  team  and  the 
program  office  and  to  determine  major  risk  drivers  in  the  program. 

To  assess  the  program’s  plans  and  risks  in  manufacturing  and  its  capacity 
to  accelerate  production  from  fiscal  years  2010  through  2015,  we  analyzed 
manufacturing  cost  and  work  performance  data  to  assess  progress  against 
plans.  We  compared  budgeted  program  labor  hours  to  actual  labor  hours, 
identified  growth  trends,  and  noted  differences  between  future  labor 
requirements  and  current  plans  to  release  engineering  staff.  We  reviewed 
data  and  briefings  provided  by  the  program  and  the  Office  of  the  Secretary 
of  Defense  (OSD)  to  assess  supplier  performance  and  ability  to  support 
accelerated  production  from  fiscal  years  2010  through  2015.  We  also 
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Appendix  I:  Scope  and  Methodology 


determined  reasons  for  manufacturing  delays,  discussed  program  and 
contractor  plans  to  improve,  and  projected  expected  impacts  on 
development  and  operational  tests.  We  also  reviewed  the  prime 
contractor’s  schedule  estimates  and  compared  them  with  relevant  best 
practices1  to  determine  the  extent  to  which  they  reflect  key  estimating 
practices  that  are  fundamental  to  having  a  reliable  schedule.  In  doing  so, 
we  interviewed  cognizant  program  officials  to  discuss  their  use  of  best 
practices  in  creating  the  program’s  current  schedule  and  interviewed 
officials  from  Naval  Air  Systems  Command  to  understand  their  approach 
and  to  obtain  results  of  their  independent  schedule  risk  analysis. 

To  assess  plans,  progress,  and  risks  in  test  activities,  we  examined 
program  documents  and  interviewed  DOD,  program  office,  and  contractor 
officials  about  current  test  plans  and  progress.  To  assess  progress  toward 
test  plans,  we  compared  the  number  of  flight  tests  conducted  as  of 
October  2008  to  those  in  the  original  test  plan  established  in  2007.  We  also 
reviewed  documents  and  interviewed  prime  contractor  officials  regarding 
flight  testing,  the  integrated  airborne  test  bed,  and  ground  testing.  To 
further  assess  the  ground  labs  and  test  bed,  we  interviewed  DOD  and 
program  officials  and  toured  the  testing  labs  and  aircraft  at  the  Lockheed 
Martin  plant  in  Fort  Worth,  Texas. 

In  performing  our  work,  we  obtained  information  and  interviewed  officials 
from  the  JSF  Joint  Program  Office,  Arlington,  Virginia;  Lockheed  Martin 
Aeronautics,  Fort  Worth,  Texas;  Defense  Contract  Management  Agency, 
Fort  Worth,  Texas;  Defense  Contract  Management  Agency,  East  Hartford, 
Connecticut;  Naval  Air  Systems  Command,  Patuxent  River,  Maryland;  Air 
Force  Operational  Test  and  Evaluation  Center,  Kirtland  Air  Force  Base, 
New  Mexico;  Air  Force  Cost  Analysis  Agency,  Arlington,  Virginia;  and  OSD 
offices  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics,  the  Director  of  Program  Analysis  and  Evaluation  and  its  Cost 
Analysis  Improvement  Group,  and  the  Director  of  Operational  Test  and 
Evaluation  in  Washington,  D.C. 

We  conducted  this  performance  audit  from  June  2008  to  March  2009  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 


^AO,  Cost  Assessment  Guide:  Best  Practices  for  Estimating  and  Managing  Program 
Costs,  Exposure  Draft,  GAO-07-1 134SP  (Washington,  D.C.:  July  2007). 


Page  32 


GAO-09-303  Joint  Strike  Fighter 


Appendix  I:  Scope  and  Methodology 


conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives. 
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Prime  Contractor’s 
Scheduling 
Management 
Processes  Meet  Many 
Best  Practices,  but 
Program  Risks  Are 
Not  Entirely  Visible 


The  success  of  any  program  depends  in  part  on  having  a  reliable  schedule 
of  when  the  program’s  set  of  work  activities  will  occur,  how  long  they  will 
take,  and  how  they  are  related  to  one  another.  As  such,  the  schedule  not 
only  provides  a  road  map  for  systematic  execution  of  a  program,  but  also 
provides  the  means  by  which  to  gauge  progress,  identify  and  address 
potential  problems,  and  promote  accountability. 

In  general,  best  practices  and  related  federal  guidance  call  for  a  program 
schedule  to  identify,  sequence,  integrate,  and  resource  all  key  activities  to 
be  performed,  and  to  understand  and  proactively  address  activities  that 
pose  critical  risks.  More  specifically,  our  research  has  identified  nine 
practices  associated  with  effective  schedule  estimating.1  These  practices 
are  (1)  capturing  all  activities,  (2)  sequencing  all  activities,  (3)  assigning 
resources  to  all  activities,  (4)  establishing  the  duration  of  all  activities, 

(5)  integrating  schedule  activities  horizontally  and  vertically, 

(6)  establishing  the  critical  path  for  all  activities,  (7)  identifying  float2 
between  activities,  (8)  conducting  a  schedule  risk  analysis,  and 

(9)  updating  the  schedule  using  logic  and  durations  to  determine  dates. 


Of  these  nine  practices,  the  JSF  program  either  met  or  partially  met  eight 
with  only  one  not  being  met.  The  area  that  did  not  meet  the  practices 
related  to  performing  a  schedule  risk  analysis.  Specifically,  the  JSF 
program  has  not  conducted  its  own  schedule  risk  analysis  that  would 
determine  the  level  of  confidence  it  has  in  meeting  completion  dates. 
Further,  an  assessment  is  also  critical  to  identifying  and  apportioning 
reserves  for  contingencies.  Since  the  JSF  program  has  not  conducted  its 
own  schedule  risk  analysis,  it  has  limited  insight  into  areas  of  risk  and 
uncertainty  in  the  schedule.  Naval  Air  Systems  Command  officials  did 
accomplish  an  independent  schedule  risk  analysis  that  indicated  that  the 
program  could  slip  more  than  2  years  based  on  the  productivity  risks 
associated  with  software  development  and  assembly  of  the  various 
airframes  as  well  as  the  time  needed  to  complete  all  flight  testing. 


In  addition  to  a  schedule  risk  analysis  not  being  performed,  we  found 
several  other  schedule  management  concerns  that  further  reduce  the 
visibility  of  manufacturing  risks.  First,  the  use  of  best  scheduling  practices 
at  the  subcontractor  level  is  still  being  developed,  potentially  affecting  the 
integration  of  subcontractor  schedules  into  the  integrated  master 


'GAO-09-3SP. 

"Float  is  the  amount  of  time  an  activity  can  slip  before  affecting  the  critical  path. 
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schedule.  Integrating  prime  and  subcontractor  schedules  is  critical  to 
meeting  program  schedules  and  cost  expectations.  The  prime  contractor  is 
working  with  subcontractors  to  increase  their  level  of  schedule  maturity. 
Another  area  of  concern  is  that  out-of-station  work  made  it  difficult  to 
identify  specific  span  times  for  individual  manufacturing  tasks.3  As  a 
result,  the  detailed  information  related  to  the  manufacturing  work  was  not 
visible  in  the  master  schedule.  Furthermore,  because  of  the  program’s 
enormous  size  and  complexity,  the  schedule  has  been  difficult  to  maintain, 
requiring  manual  validation  processes  to  ensure  its  integrity  and  validity. 
Ongoing  JSF  schedule  validity  will  be  an  area  that  needs  careful  attention 
as  it  represents  a  potential  weak  point  in  the  overall  implementation  of  the 
integrated  master  schedule. 

Despite  this  shortcoming,  it  is  also  important  to  recognize  the  significant 
progress  that  the  JSF  program  team  has  made  in  the  area  of  schedule 
management.  Since  the  previous  Defense  Contract  Management  Agency 
schedule  review,  both  the  schedule  and  the  processes  to  manage  it  have 
greatly  improved.  For  example,  the  schedule  can  track  and  verify  activities 
associated  with  over  600  projects.  It  also  successfully  captures  and 
sequences  key  activities  and  establishes  the  critical  path  between  key 
activities,  allowing  the  program  to  examine  the  impacts  of  schedule  delays 
and  determine  schedule  flexibility. 


The  Sheer  Size  and 
Complexity  of  the  JSF 
Schedule  Have  Created 
Major  Challenges  to 
Ensuring  Schedule 
Integrity  and  Validity 


The  JSF  schedule  is  maintained  in  Microsoft  Project  and  consists  of  over 
600  individual  projects.  Because  the  size  and  complexity  of  the  schedule  is 
so  immense,  it  has  been  difficult  to  maintain.  As  such,  a  number  of  manual 
validation  processes  are  required  to  ensure  its  integrity  and  validity.  To  its 
credit,  the  contractor  has  developed  custom  processes  and  tools  to  help 
manage  the  program  schedule.  However,  because  of  its  enormous  size  and 
complexity,  the  JSF’s  ongoing  schedule  validity  will  be  an  area  that  needs 
careful  attention  as  it  represents  a  potential  weak  point  in  the  overall 
implementation  of  the  integrated  master  schedule. 

Because  the  schedule  was  so  large,  we  reviewed  a  subset  of  it,  focusing  on 
the  delivery  of  one  airframe  for  each  variant  of  the  F-35  being  produced 
(i.e.,  BF4,  AF1,  and  AF3).  This  subset  schedule  covered  a  time  span  from 


3It  was  difficult  to  see  specific  span  times  for  individual  manufacturing  tasks  because  work 
that  did  not  finish  in  its  designated  station  was  carried  forward  and  completed  at  a 
different  station  down  the  production  line.  This  work  was  called  out-of-station. 
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August  2006  through  September  2014,  and  we  analyzed  it  against  our  best 
practices  for  effective  schedule  estimating.  See  table  6  for  the  results  of 
our  analyses  relative  to  each  of  the  nine  practices. 


Table  6:  Prime  Contractor  Schedule  Management  System  Compared  to  Best  Practices 


Criterion  Explanation 


Criterion  met?3  GAO  analysis 


Capturing  all  activities  The  schedule  should  reflect  all  activities 
(e.g.,  steps,  events,  outcomes,  etc.)  as 
defined  in  the  program's  work  breakdown 
structure,  to  include  activities  to  be 
performed  by  both  the  government  and 
its  contractors. 


Yes  The  program's  schedule  reflected  both 

government  and  contractor  activities, 
including  activities  related  to  designing, 
developing,  testing,  and  manufacturing  the 
three  variants  of  the  F -35. 


Sequencing  all  activities  The  schedule  should  be  planned  so  that  it 
can  meet  program-critical  dates.  To  meet 
this  objective,  key  activities  need  to  be 
logically  sequenced  in  the  order  that  they 
are  to  be  carried  out.  In  particular, 
activities  that  must  finish  prior  to  the  start 
of  other  activities  (i.e.,  predecessor 
activities)  as  well  as  activities  that  cannot 
begin  until  other  activities  are  completed 
(i.e.,  successor  activities)  should  be 
identified.  By  doing  so,  interdependencies 
among  activities  that  collectively  lead  to 
the  accomplishment  of  events  or 
milestones  can  be  established  and  used 
as  a  basis  for  guiding  work  and 
measuring  progress. 


Yes  The  schedule  included  the  sequencing  of  key 

activities,  meaning  that  it  included  both  the 
predecessor  and  successor  activities  and 
thus  established  interdependencies  among 
the  activities  that  form  the  basis  for  guiding 
work  and  measuring  progress.  The  schedule 
logic  was  validated  by  numerous  metrics  that 
were  tracked  on  a  monthly  basis.  For 
example,  the  contractor  collected  metrics 
that  verified  that  logic  was  not  maintained  at 
summary-level  tasks,  activities  had  both  a 
predecessor  and  successor  relationship,  and 
external  links  were  valid.  Additionally,  the 
metrics  tracked  the  use  of  hard  constraints 
so  if  any  were  found,  there  were  procedures 
to  remove  any  constraints  that  override 
schedule  logic.  These  detailed  metrics 
provided  assurance  in  the  validity  of  the 
schedule.  They  also  enabled  the  program  to 
have  confidence  that  the  sequencing  of 
activities  was  not  being  overwritten  by  an 
overuse  of  constraints. 
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Criterion  Explanation 

Assigning  resources  to  The  schedule  should  reflect  what 
all  activities  resources  (i.e.,  labor,  material,  and 

overhead)  are  needed  to  do  the  work, 
whether  all  required  resources  will  be 
available  when  they  are  needed,  and 
whether  any  funding  or  time  constraints 
exist. 


Criterion  met?3  GAO  analysis 

Partially  The  resources,  labor,  materials,  and 

overhead  were  not  loaded  directly  into  the 
M  icrosoft  P  reject  schedules.  Instead,  the 
budgets  were  maintained  in  another  system, 
which  was  integrated  with  both  the  schedule 
and  the  Manufacturing  Resource  Planning 
(MRP)  system.  While  not  technically 
assigned  to  individual  activities,  the 
resources  were  integrated  into  the 
management  of  the  system  through  the  work 
packages  identified  in  the  earned  value 
system.  The  manufacturing  activity  resources 
were  maintained  in  the  MRP  system  and  the 
schedule  reflected  the  level  of  work 
accomplished  in  various  production  stations 
based  on  the  status  of  the  individual  work 
cards.  Program  officials  provided  a  data 
trace  between  the  schedule,  MRP  system, 
and  earned  value  system  to  demonstrate 
how  the  resources  were  integrated  with  the 
schedule. 


Establishing  the  duration  The  schedule  should  realistically  reflect 
of  all  activities  how  long  each  activity  will  take  to 

execute.  In  determining  the  duration  of 
each  activity,  the  same  rationale,  data, 
and  assumptions  used  for  cost  estimating 
should  be  used  for  schedule  estimating. 
Further,  these  durations  should  be  as 
short  as  possible  and  they  should  have 
specific  start  and  end  dates.  Excessively 
long  periods  needed  to  execute  an 
activity  should  prompt  further 
decomposition  of  the  activity  so  that 
shorter  execution  durations  will  result. 


Partially  The  schedule  established  the  durations  of 

key  activities  based  on  historical  data  from 
the  contractor's  experience,  projected 
savings  to  be  gained  from  implementing 
robust  processes  to  maintain  the  flow  of  work 
on  the  manufacturing  floor,  and  expected 
savings  from  investing  up  front  in  modeling 
and  simulation.  The  program  tracked  the 
actual  start,  progress,  and  actual  finish  of 
activities  in  the  schedule,  allowing  for 
collection  of  trending  information.  Progress 
metrics  were  tracked  by  the  contractor  in  the 
status  metric  process  and  reported  to  the 
program,  enabling  tracking  against  the 
schedules. 

It  was  difficult,  however,  to  see  the  specific 
span  times  for  individual  manufacturing  tasks 
because  work  that  did  not  finish  in  its 
designated  station  was  carried  forward  and 
completed  at  a  different  station  down  the 
production  line  (this  work  was  called  out-of- 
station).  As  a  result,  the  detailed  information 
related  to  the  manufacturing  work  was 
maintained  in  the  MRP  system,  butwas  not 
visible  within  the  schedule  itself. 

Nonetheless,  the  contractor  demonstrated 
the  relationship  between  the  specific  work 
cards  in  the  MRP  system  and  how  those 
data  translated  to  the  schedule. 
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Criterion  Explanation 


Criterion  met?3  GAO  analysis 


Integrating  schedule  The  schedule  should  be  horizontally  Yes 
activities  horizontally  and  integrated,  meaning  that  it  should  link  the 
vertically  products  and  outcomes  associated  with 

already  sequenced  activities.  These  links 
are  commonly  referred  to  as  handoffs 
and  serve  to  verify  that  activities  are 
arranged  in  the  right  order  to  achieve 
aggregated  products  or  outcomes.  The 
schedule  should  also  be  vertically 
integrated,  meaning  that  traceability 
exists  among  varying  levels  of  activities 
and  supporting  tasks  and  subtasks.  Such 
mapping  or  alignment  among  levels 
enables  different  groups  to  work  to  the 
same  master  schedule. 


Establishing  the  critical  Using  scheduling  software  the  critical  Yes 
path  for  all  activities  path— the  longest  duration  path  through 
the  sequenced  list  of  key  activities — 
should  be  identified.  The  establishment  of 
a  program's  critical  path  is  necessary  for 
examining  the  effects  of  any  activity 
slipping  along  this  path.  Potential 
problems  that  may  occur  on  or  near  the 
critical  path  should  also  be  identified  and 
reflected  in  the  scheduling  of  the  time  for 
high-risk  activities. 


The  schedule  was  both  horizontally  and 
vertically  integrated  and  there  was  a  robust 
process  in  place  to  verify  the  traceability  at 
each  progress  cycle.  The  contractor's  project 
status  sheet  identified  where  activities  were 
missing  relationships,  enabling  a  critical  path 
to  be  derived  by  the  tool's  critical  path 
analysis.  Relationships  were  maintained 
across  the  600+  individual  projects  and 
verified  through  periodic  schedule  health 
checks.  The  contractor  provided  report 
outputs  and  demonstrated  the  process  it 
used  to  identify  and  validate  that  the  links 
across  projects  were  intact.  Similarly,  for 
vertical  integration  the  contractor 
demonstrated  how  status  was  rolled  up  from 
the  individual  project  files  to  the  proper 
higher-level  summary  activities  so  the  overall 
schedule  reflected  the  proper  dates  and 
status. 

An  area  of  concern  was  with  the  integration 
of  the  subcontractors'  schedules  into  the 
integrated  master  schedule.  The  issue  was 
not  as  much  the  integration  of  the  data,  but 
the  use  of  scheduling  best  practices  at  the 
subcontractor  level.  The  contractor  was 
working  with  its  subcontractors  to  increase 
the  level  of  scheduling  maturity.  Moreover, 
given  the  massive  size  of  the  integrated 
master  schedule  and  number  of  files  being 
utilized  to  integrate  it,  this  was  an  area  that 
needed  continual  monitoring  to  verify 
compliance. 

The  program's  critical  path  was  defined  using 
scheduling  software.  Critical  paths  were 
defined  by  individual  airframes  and  were  also 
calculated  at  the  total  program  level.  The 
contractor  provided  output  from  the  critical 
path  analysis  and  demonstrated  the  process 
it  used  to  produce  the  critical  path.  Given  the 
large  number  of  files  and  overall  size  of  the 
integrated  master  schedule,  it  was  not 
possible  to  validate  the  critical  path  through 
independentanalysis. 

The  contractor's  metric  process  performed 
during  each  status  cycle  helped  to  maintain 
and  validate  the  critical  path  produced  by  the 
tool.  In  the  subset  schedule  we  reviewed 
there  were  no  hard  constraints,  that  is,  we 
found  no  "must  start"  or  "must  finish"  on 
dates  that  could  override  the  critical  path 
analysis  and  cause  invalid  results. 
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Criterion  Explanation 

Identifying  float  between  The  schedule  should  identify  float— the 
activities  time  that  a  predecessor  activity  can  slip 

before  the  delay  affects  successor 
activities— so  that  schedule  flexibility  can 
be  determined.  As  a  general  rule, 
activities  along  the  critical  path  typically 
have  the  least  amount  of  float. 


Criterion  met?3  GAO  analysis 

Yes  The  contractor's  overall  schedule  process 

enabled  good  visibility  into  the  float  between 
activities  and  demonstrated  thatfloatis 
actively  managed.  The  metrics  tracked  by 
the  contractor  clearly  identified  where  there 
were  issues  with  the  amount  of  float  on  an 
activity,  including  which  tasks  had  negative 
float  The  contractor  had  defined  processes 
fortracing  and  understanding  what  was 
driving  that  float.  In  addition,  the  tracking  and 
removal  of  hard  constraints  in  the  schedule 
enabled  the  tool  to  generate  the  float  via  the 
critical  path  analysis. 


Conducting  a  schedule  A  schedule  risk  analysis  uses  a  good 
risk  analysis  critical  path  method  schedule  and  data 

about  project  schedule  risks  as  well  as 
Monte  Carlo  simulation  techniques  to 
predict  the  level  of  confidence  in  meeting 
a  program's  completion  date,  the  amount 
of  time  contingency  needed  for  a  level  of 
confidence,  and  the  identification  of  high- 
priority  risks.  This  analysis  focuses  not 
only  on  critical  path  activities  but  also  on 
other  schedule  paths  that  may  become 
critical.  A  schedule/cost  risk  assessment 
recognizes  the  interrelationship  between 
schedule  and  costand  captures  the  risk 
that  schedule  durations  and  cost 
estimates  may  vary  because  of,  among 
other  things,  limited  data,  optimistic 
estimating,  technical  challenges,  and  lack 
of  qualified  personnel.  As  a  result,  the 
baseline  schedule  should  include  a  buffer 
ora  reserve  of  extra  time.  Schedule 
reserve  for  contingencies  should  be 
calculated  by  performing  a  schedule  risk 
analysis.  As  a  general  rule,  the  reserve 
should  be  held  by  the  project  manager 
and  applied  as  needed  to  those  activities 
that  take  longer  than  scheduled  because 
of  the  identified  risks.  Reserves  of  time 
should  not  be  apportioned  in  advance  to 
any  specific  activity  since  the  risks  that 
will  actually  occur  and  the  magnitude  of 
their  impact  is  not  known  in  advance. 


No  The  contractor  did  not  perform  a  schedule 

risk  analysis  that  would  have  determined  the 
level  of  confidence  in  meeting  the  program's 
completion  date  and  identified  reserves  for 
contingencies  because  it  did  not  have  time  to 
do  this  analysis.  The  contractor  spent  much 
of  the  past  year  rebaselining  its  schedule  so 
it  would  be  more  realistic  and  meet  best 
practices  and  said  that  it  intended  to  do 
schedule  risk  analyses  in  the  future. 

Naval  Air  Systems  Command  conducted  an 
independentschedule  risk  analysis  that 
indicated  that  the  program  could  slip  more 
than  2  years  based  on  productivity  risks 
associated  with  software  development  and 
assembly  of  the  various  airframes  as  well  as 
the  time  needed  to  successfully  complete  all 
flight  testing.  Based  on  the  command's 
results,  the  contractor  does  not  have  an 
adequate  amount  of  schedule  reserve  to 
mitigate  testing  and  production  risks.  This 
low  reserve  means  that  any  delays  in  these 
areas  would  have  an  immediate  impact  to 
the  overall  project  completion  date. 

Moreover,  there  was  concern  from  outside 
experts  that  savings  gained  from 
implementing  robust  manufacturing 
processes  and  relying  more  on  modeling  and 
simulation  may  not  be  achieved  as  they 
exceed  anything  done  before.  Both  the 
contractor  and  thej  SF  Program  Office 
disputed  the  findings  of  the  command's 
schedule  risk  analysis  primarily  because  the 
analysis  was  done  without  direct  involvement 
of  program  officials. 
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Criterion 


Explanation 


Criterion  met?3  GAO  analysis 


Updating  the  schedule  The  schedule  should  use  logic  and 
using  logic  and  durations  durations  in  order  to  reflect  realistic  start 
to  determine  the  dates  and  completion  dates  for  program 
activities.  The  schedule  should  be 
continually  monitored  to  determine  when 
forecasted  completion  dates  differ  from 
the  planned  dates,  which  can  be  used  to 
determine  whether  schedule  variances 
will  affect  downstream  work.  Maintaining 
the  integrity  of  the  schedule  logic  is  not 
only  necessary  to  reflect  true  status,  but 
is  also  required  before  conducting  a 
schedule  risk  analysis.  The  schedule 
should  avoid  logic  overrides  and  artificial 
constraint  dates  that  are  chosen  to  create 
a  certain  result  on  paper.  T o  ensure  that 
the  schedule  is  properly  updated, 
individuals  trained  in  critical  path  method 
scheduling  should  be  responsible  for 
updating  the  schedule  status. 


Y es  The  contractor  demonstrated  that  it  relies  on 

logic  and  duration  to  determine  the  dates 
when  it  updates  the  status  of  its  schedule. 

F or  example,  the  contractor  showed  us  how 
it  monitors  the  schedule  on  a  weekly  basis  to 
ensure  that  start  and  completion  dates  are 
realistic  and  to  identify  where  there  are 
variances  between  planned  and  actual 
progress.  The  contractor  also  demonstrated 
how  it  verifies  that  all  hard  constraints  have 
been  removed  so  thatthe  schedule  depicts 
actual  conditions.  Moreover,  the  personnel 
responsible  for  maintaining  the  schedule 
demonstrated  a  strong  understanding  of 
critical  path  method  scheduling,  adding  to 
our  confidence  that  the  schedule  is  being 
properly  updated  in  accordance  with  best 
practices. 


Source:  GAO  analysis  of  DOD  data. 

“"Yes"  means  thatthe  program  provided  documentation  demonstrating  satisfaction  of  the  criterion. 
"Partial"  means  thatthe  program  provided  documentation  demonstrating  satisfaction  of  part  of  the 
criterion.  "No"  means  thatthe  program  has  yet  to  provide  documentation  demonstrating  satisfaction  of 
the  criterion. 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 
3000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3000 


ACQUISITION 

TECHNOLOGY  ,  , 

AND  LOGISTICS  WAR  -  5  2009 

Mr.  Michael  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  N.W. 

Washington,  DC  20548 

Dear  Mr.  Sullivan: 


This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  draft  report  09- 
303,  “Joint  Strike  Fighter:  DOD’S  Proposal  to  Accelerate  Procurement  before 
Completing  Development  Increases  the  Government’s  Financial  Risk”  dated  February  2. 
2009,  (GAO  Code  120742).  Detailed  comments  on  the  report  recommendations  are 
enclosed. 

The  DoD  partially  concurs  with  recommendation  one  and  concurs  with 
recommendation  two.  The  rationale  for  our  position  is  included  in  the  enclosure. 

We  appreciate  the  opportunity  to  comment  on  the  draft  report.  My  point  of 
contact  for  this  effort  is  CAPT  Gregg  Sears,  703-695-3015,  gregg.sears@osd.mil. 


Sincerely, 


Portfolio  Systems  Acquisition 

Enclosure: 

As  stated 
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GAO  Draft  Report  Dated  FEBRUARY  2, 2009 
GAO-09-303  (GAO  CODE  120742) 


“JOINT  STRIKE  FIGHTER:  DOD’S  PROPOSAL  TO  ACCELERATE 
PROCUREMENT  BEFORE  COMPLETING  DEVELOPMENT  INCREASES  THE 
GOVERNMENT’S  FINANCIAL  RISK 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 


RECOMMENDATION  1:  The  GAO  recommends  that  the  Secretary  of  Defense  direct  that  the 
Joint  Strike  Fighter  program  office  report  to  the  Congressional  Defense  Committees  by 
October  1 , 2009. 

POD  RESPONSE:  Partially  concur  The  Department  agrees  that  a  report  to  the  Congressional 
Defense  Committees  explaining  the  Department’s  plan  to  transition  from  cost  reimbursable 
contracts  to  fixed  price  contracts  for  Joint  Strike  Fighter  (JSF)  procurement  would  be  beneficial 
to  enhance  Congressional  oversight  and  confidence  in  the  JSF  program.  However,  the 
Department  believes  that  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  (USD  AT&L)  should  be  responsible  for  the  report,  not  the  Joint  Strike  Fighter 
program.  As  the  Milestone  Decision  Authority  (MDA),  USD  (AT&L)  is  responsible  for 
approval  of  the  JSF  Acquisition  Strategy,  including  the  contracting  strategy,  as  well  as  approval 
of  individual  Low  Rate  Initial  Production  (LRIP)  contract  awards  and  the  transition  to  Full  Rate 
Production  (FRP).  The  JSF  Acquisition  Strategy,  utilizing  cost  reimbursable  contracts  that 
contain  cost  and  performance  incentives  for  LRIP,  has  been  approved  by  the  Department  and 
involved  a  review  of  the  associated  risks,  benefits,  and  options.  The  USD  AT&L,  on  behalf  of 
the  Secretary  will  provide  a  report  to  the  Congressional  Defense  Committees  by  October  1 ,  2009, 
that  addresses  the  requirements  in  the  recommendation. 

The  Department  would  also  like  to  state  that  since  the  FY  2010  President’s  budget  had 
not  yet  been  submitted  to  Congress  at  the  time  of  the  draft  report,  the  fact  is  that  much  of  the  data 
that  the  GAO  used  as  the  basis  of  their  report  and  to  support  its  recommendation  was  pre- 
decisional.  The  JSF  program  of  record  remained  as  submitted  in  the  FY  2009  President’s 
Budget,  adjusted  by  the  FY  2009  Defense  Appropriations  Bill.  The  Department  believes  that  the 
GAO’s  use  of  pre-decisional  budget  information  to  support  its  findings  is  largely  conjectural,  and 
likely  to  create  confusion  if  the  final  FY  2010  program  of  record  is  different  than  what  the  GAO 
reported. 

The  JSF  program  does  have  substantial  concurrency  of  development,  test  and  production 
built  into  the  JSF  schedule,  a  fact  that  the  Department  acknowledges  and  has  approved.  That 
concurrency  is  designed  to  provide  the  warfighters  with  a  5lh  generation  strike  fighter  to  replace 
aging  legacy  aircraft  as  quickly  as  possible,  as  efficiently  as  possible,  and  as  affordably  as 
possible.  The  Department  acknowledges  the  risks,  and  benefits,  of  the  concurrency  required  to 
acquire  and  deliver  this  critical  capability.  The  Department  also  believes  that  the  program  is  well 
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managed,  with  the  proper  amount  of  oversight,  and  well  positioned  to  succeed  in  accomplishing 
this  mission. 

RECOMMENDATION  2:  The  GAO  recommends  that  the  Secretary  of  Defense  direct  that  the 
Joint  Program  Office  ensure  the  prime  contractor  performs  periodic  schedule  risk  analyses  for 
the  Joint  Strike  Fighter  (JSF)  program  to  provide  better  insight  into  Management  Reserve, 
production  efficiencies,  and  schedule  completion  to  allow  for  corrections  as  early  as  possible. 

POD  RESPONSE:  Concur  The  Department  will  direct  that  the  contractor  perform  periodic 
schedule  risk  analysis.  The  Department  will  coordinate  with  the  JSF  Program  Office  and 
determine  an  optimum  schedule  for  the  contractor  to  meet  that  will  provide  appropriate  insight 
into  JSF  cost  and  schedule  to  influence  key  program  milestones  and  decision  making. 
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GAO  Contact 


Michael  Sullivan  (202)  512-4841  or  sullivanm@gao.gov 


Acknowledgments 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  Web  site  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 

MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngcl@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 

